Task 12738047

Name	hadcm3n_o2e3_1900_40_007198430_0
Workunit	7396710
Created	28 Mar 2011, 14:03:24 UTC
Sent	1 Apr 2011, 2:06:00 UTC
Report deadline	1 Jul 2011, 9:33:11 UTC
Received	17 May 2011, 22:27:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	615938
Run time	15 days 5 hours 45 min 51 sec
CPU time	14 days 14 hours 37 min 12 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	2.45 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:27:25 (74124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:27 (74124): No heartbeat from core client for 30 sec - exiting 12:27:28 (74124): No heartbeat from core client for 30 sec - exiting 12:27:29 (74124): No heartbeat from core client for 30 sec - exiting 12:27:30 (74124): No heartbeat from core client for 30 sec - exiting 12:27:31 (74124): No heartbeat from core client for 30 sec - exiting 12:27:32 (74124): No heartbeat from core client for 30 sec - exiting 12:27:33 (74124): No heartbeat from core client for 30 sec - exiting 12:27:34 (74124): No heartbeat from core client for 30 sec - exiting 12:27:35 (74124): No heartbeat from core client for 30 sec - exiting 12:27:36 (74124): No heartbeat from core client for 30 sec - exiting 12:27:37 (74124): No heartbeat from core client for 30 sec - exiting 12:27:38 (74124): No heartbeat from core client for 30 sec - exiting 12:27:39 (74124): No heartbeat from core client for 30 sec - exiting 12:27:40 (74124): No heartbeat from core client for 30 sec - exiting 12:27:41 (74124): No heartbeat from core client for 30 sec - exiting 12:27:42 (74124): No heartbeat from core client for 30 sec - exiting 12:27:43 (74124): No heartbeat from core client for 30 sec - exiting 12:27:44 (74124): No heartbeat from core client for 30 sec - exiting 12:27:45 (74124): No heartbeat from core client for 30 sec - exiting 12:27:46 (74124): No heartbeat from core client for 30 sec - exiting 12:27:47 (74124): No heartbeat from core client for 30 sec - exiting 12:27:48 (74124): No heartbeat from core client for 30 sec - exiting 12:27:49 (74124): No heartbeat from core client for 30 sec - exiting 12:27:50 (74124): No heartbeat from core client for 30 sec - exiting 12:27:51 (74124): No heartbeat from core client for 30 sec - exiting 12:27:52 (74124): No heartbeat from core client for 30 sec - exiting 12:27:53 (74124): No heartbeat from core client for 30 sec - exiting 12:27:54 (74124): No heartbeat from core client for 30 sec - exiting 12:27:55 (74124): No heartbeat from core client for 30 sec - exiting 12:27:56 (74124): No heartbeat from core client for 30 sec - exiting 12:27:57 (74124): No heartbeat from core client for 30 sec - exiting 12:27:58 (74124): No heartbeat from core client for 30 sec - exiting 12:27:59 (74124): No heartbeat from core client for 30 sec - exiting 12:28:00 (74124): No heartbeat from core client for 30 sec - exiting 12:28:01 (74124): No heartbeat from core client for 30 sec - exiting 12:28:02 (74124): No heartbeat from core client for 30 sec - exiting 12:28:03 (74124): No heartbeat from core client for 30 sec - exiting 12:28:04 (74124): No heartbeat from core client for 30 sec - exiting 12:28:05 (74124): No heartbeat from core client for 30 sec - exiting 12:28:06 (74124): No heartbeat from core client for 30 sec - exiting 12:28:07 (74124): No heartbeat from core client for 30 sec - exiting 12:28:08 (74124): No heartbeat from core client for 30 sec - exiting 12:28:09 (74124): No heartbeat from core client for 30 sec - exiting 12:28:10 (74124): No heartbeat from core client for 30 sec - exiting 12:28:11 (74124): No heartbeat from core client for 30 sec - exiting 12:28:12 (74124): No heartbeat from core client for 30 sec - exiting 12:28:13 (74124): No heartbeat from core client for 30 sec - exiting 12:28:14 (74124): No heartbeat from core client for 30 sec - exiting 12:28:15 (74124): No heartbeat from core client for 30 sec - exiting 12:28:16 (74124): No heartbeat from core client for 30 sec - exiting 12:28:17 (74124): No heartbeat from core client for 30 sec - exiting 12:28:18 (74124): No heartbeat from core client for 30 sec - exiting 12:28:19 (74124): No heartbeat from core client for 30 sec - exiting 12:28:20 (74124): No heartbeat from core client for 30 sec - exiting 12:28:21 (74124): No heartbeat from core client for 30 sec - exiting 12:28:22 (74124): No heartbeat from core client for 30 sec - exiting 12:28:23 (74124): No heartbeat from core client for 30 sec - exiting 12:28:24 (74124): No heartbeat from core client for 30 sec - exiting 12:28:25 (74124): No heartbeat from core client for 30 sec - exiting 12:28:26 (74124): No heartbeat from core client for 30 sec - exiting 12:28:27 (74124): No heartbeat from core client for 30 sec - exiting 12:28:28 (74124): No heartbeat from core client for 30 sec - exiting 12:28:29 (74124): No heartbeat from core client for 30 sec - exiting 12:28:30 (74124): No heartbeat from core client for 30 sec - exiting 12:28:31 (74124): No heartbeat from core client for 30 sec - exiting 12:28:32 (74124): No heartbeat from core client for 30 sec - exiting 12:28:33 (74124): No heartbeat from core client for 30 sec - exiting 12:28:34 (74124): No heartbeat from core client for 30 sec - exiting 12:28:35 (74124): No heartbeat from core client for 30 sec - exiting 12:28:36 (74124): No heartbeat from core client for 30 sec - exiting 12:28:37 (74124): No heartbeat from core client for 30 sec - exiting 12:28:38 (74124): No heartbeat from core client for 30 sec - exiting 12:28:39 (74124): No heartbeat from core client for 30 sec - exiting 12:28:40 (74124): No heartbeat from core client for 30 sec - exiting 12:28:41 (74124): No heartbeat from core client for 30 sec - exiting 12:28:42 (74124): No heartbeat from core client for 30 sec - exiting 12:28:43 (74124): No heartbeat from core client for 30 sec - exiting 12:28:44 (74124): No heartbeat from core client for 30 sec - exiting 12:28:45 (74124): No heartbeat from core client for 30 sec - exiting 12:28:46 (74124): No heartbeat from core client for 30 sec - exiting 12:28:47 (74124): No heartbeat from core client for 30 sec - exiting 12:28:48 (74124): No heartbeat from core client for 30 sec - exiting 12:28:49 (74124): No heartbeat from core client for 30 sec - exiting 12:28:50 (74124): No heartbeat from core client for 30 sec - exiting 12:28:51 (74124): No heartbeat from core client for 30 sec - exiting 12:28:52 (74124): No heartbeat from core client for 30 sec - exiting 12:28:53 (74124): No heartbeat from core client for 30 sec - exiting 12:28:54 (74124): No heartbeat from core client for 30 sec - exiting 12:28:55 (74124): No heartbeat from core client for 30 sec - exiting 12:28:56 (74124): No heartbeat from core client for 30 sec - exiting 12:28:57 (74124): No heartbeat from core client for 30 sec - exiting 12:28:58 (74124): No heartbeat from core client for 30 sec - exiting 12:47:43 (39330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:54:51 (245): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:14:45 (221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:56:24 (229): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:00:27 (230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:47:38 (246): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x4801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x4801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(275,0xa09fb540) malloc: * error for object 0x7801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 16:07:04 (223): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:23:39 (285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:30:00 (245): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... execl(/Volumes/Totoro/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 132775) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5675, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 May 2011 21:26:28	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	622,080	1,260,127	2.0257
17 May 2011 06:03:13	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	596,160	1,207,846	2.0260
16 May 2011 13:40:39	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	570,240	1,155,211	2.0258
15 May 2011 22:26:52	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	544,320	1,102,608	2.0257
15 May 2011 07:16:16	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	518,400	1,050,255	2.0260
14 May 2011 16:11:44	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	492,480	997,716	2.0259
14 May 2011 01:04:12	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	466,560	945,179	2.0258
13 May 2011 09:29:50	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	440,640	892,176	2.0247
12 May 2011 17:59:40	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	414,720	839,487	2.0242
12 May 2011 02:47:52	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	388,800	786,900	2.0239
11 May 2011 11:21:57	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	362,880	734,231	2.0233
10 May 2011 19:44:59	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	336,960	681,673	2.0230
10 May 2011 03:12:30	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	311,040	628,979	2.0222
09 May 2011 10:53:01	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	285,120	576,500	2.0220
08 May 2011 19:35:47	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	259,200	523,708	2.0205
08 May 2011 04:06:45	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	233,280	471,182	2.0198
07 May 2011 12:55:17	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	207,360	418,606	2.0187
06 May 2011 21:49:06	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	181,440	366,465	2.0198
06 May 2011 06:52:34	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	155,520	314,703	2.0236
05 May 2011 15:50:49	615938	12738047	hadcm3n_o2e3_1900_40_007198430_0	129,600	262,873	2.0283