Task 13023518

Name	hadcm3n_t3ow_1940_40_007315293_1
Workunit	7512723
Created	28 Jun 2011, 20:41:22 UTC
Sent	28 Jun 2011, 20:52:35 UTC
Report deadline	28 Sep 2011, 4:19:46 UTC
Received	10 Sep 2011, 15:44:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153671
Run time	6 days 7 hours 21 min 15 sec
CPU time	5 days 15 hours 58 min 20 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x819204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x819200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x819204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x819200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x201a200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x801404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3319,0xa041a540) malloc: * error for object 0x801400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:44:48 (7474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:19:45 (10321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:51:19 (12706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:58 (13711): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:24 (16263): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:56:19 (17267): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:39 (99309): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 137415) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1796, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2011 15:42:18	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	518,400	485,902	0.9373
08 Jul 2011 07:38:30	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	492,480	461,502	0.9371
08 Jul 2011 00:00:40	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	466,560	437,091	0.9368
07 Jul 2011 17:49:10	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	440,640	413,436	0.9383
07 Jul 2011 17:49:09	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	414,720	389,155	0.9384
07 Jul 2011 17:49:08	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	388,800	364,761	0.9382
07 Jul 2011 17:49:08	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	362,880	340,601	0.9386
07 Jul 2011 17:49:08	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	336,960	316,171	0.9383
05 Jul 2011 01:45:49	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	311,040	291,745	0.9380
04 Jul 2011 17:07:20	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	285,120	267,364	0.9377
04 Jul 2011 09:13:41	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	259,200	242,983	0.9374
03 Jul 2011 23:01:41	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	233,280	218,568	0.9369
03 Jul 2011 11:51:09	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	207,360	194,053	0.9358
03 Jul 2011 04:19:32	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	181,440	169,603	0.9348
02 Jul 2011 20:38:11	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	155,520	145,079	0.9329
02 Jul 2011 00:15:16	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	129,600	120,752	0.9317
30 Jun 2011 16:15:20	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	103,680	96,552	0.9313
30 Jun 2011 07:59:15	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	77,760	72,527	0.9327
29 Jun 2011 19:18:54	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	51,840	48,488	0.9353
29 Jun 2011 07:12:39	1153671	13023518	hadcm3n_t3ow_1940_40_007315293_1	25,920	24,162	0.9322