climateprediction.net home page
Task 16021063

Task 16021063

Name hadcm3n_3lkz_1980_40_008371213_2
Workunit 8522072
Created 17 Sep 2013, 13:03:29 UTC
Sent 17 Sep 2013, 13:07:09 UTC
Report deadline 17 Dec 2013, 20:34:20 UTC
Received 13 Oct 2013, 20:44:49 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1169903
Run time 15 days 0 hours 57 min 12 sec
CPU time 11 days 3 hours 0 min 47 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 3.15 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
23:22:24 (7292): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:03:51 (17370): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
01:29:40 (81613): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
01:39:40 (36279): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
17:24:18 (79329): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:37:24 (14091): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:24:09 (25998): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:59:08 (54734): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:32:04 (59540): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:55:10 (37588): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x104c004: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x104c000: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x1030a04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x681f604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x81f604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x801f604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x801f600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(41964,0xa05ca540) malloc: *** error for object 0x6000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
03:51:49 (41964): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:53:02 (59992): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:45:21 (61292): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:58:16 (74152): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:29:49 (91255): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:59:24 (36794): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:00:59 (42560): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:46:20 (50603): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
07:30:09 (81188): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
00:04:33 (39312): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
03:49:16 (751): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
13:29:28 (9301): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3lkz_1980_40_008371213/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
13 Oct 2013 18:34:27 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 518,400 961,274 1.8543
13 Oct 2013 02:16:52 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 492,480 913,805 1.8555
12 Oct 2013 11:33:28 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 466,560 866,504 1.8572
11 Oct 2013 18:58:03 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 440,640 819,169 1.8590
10 Oct 2013 09:16:28 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 414,720 771,465 1.8602
08 Oct 2013 23:43:28 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 388,800 722,815 1.8591
07 Oct 2013 10:46:41 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 362,880 673,786 1.8568
06 Oct 2013 15:13:58 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 336,960 625,467 1.8562
05 Oct 2013 19:02:42 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 311,040 576,216 1.8525
04 Oct 2013 22:49:24 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 285,120 527,436 1.8499
04 Oct 2013 03:18:25 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 259,200 478,817 1.8473
03 Oct 2013 06:25:10 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 233,280 429,743 1.8422
02 Oct 2013 11:41:58 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 207,360 381,602 1.8403
01 Oct 2013 15:17:26 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 181,440 333,076 1.8357
30 Sep 2013 19:29:08 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 155,520 285,041 1.8328
30 Sep 2013 00:26:26 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 129,600 236,855 1.8276
29 Sep 2013 05:22:22 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 103,680 188,215 1.8153
27 Sep 2013 02:45:17 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 77,760 140,068 1.8013
26 Sep 2013 08:32:07 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 51,840 92,225 1.7790
25 Sep 2013 09:29:45 1169903 16021063 hadcm3n_3lkz_1980_40_008371213_2 25,920 45,205 1.7440


©2024 climateprediction.net