Task 13289168

Name	hadcm3n_p42s_1940_40_007420625_1
Workunit	7618260
Created	24 Aug 2011, 23:30:52 UTC
Sent	24 Aug 2011, 23:31:15 UTC
Report deadline	24 Nov 2011, 6:58:26 UTC
Received	30 Nov 2011, 16:44:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1006750
Run time	27 days 15 hours 32 min 33 sec
CPU time	26 days 21 hours 32 min 47 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	1.85 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(145,0xa0862540) malloc: * error for object 0x801004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(145,0xa0862540) malloc: * error for object 0x801000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=145, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( SIGSEGV: segmentation violation SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(81715,0xa0862540) malloc: * error for object 0x811a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81715,0xa0862540) malloc: * error for object 0x800804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81715,0xa0862540) malloc: * error for object 0x800800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(81715,0xa0862540) malloc: * error for object 0x812a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=81715, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(148,0xa0862540) malloc: * error for object 0x801a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(148,0xa0862540) malloc: * error for object 0x800804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(148,0xa0862540) malloc: * error for object 0x800800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(148,0xa0862540) malloc: * error for object 0x802a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=148, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Sep 2011 03:38:53	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	855,360	2,261,789	2.6443
28 Sep 2011 08:20:15	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	829,440	2,194,653	2.6459
26 Sep 2011 23:57:13	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	803,520	2,125,005	2.6446
26 Sep 2011 03:11:03	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	777,600	2,056,311	2.6444
25 Sep 2011 00:45:20	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	751,680	1,986,077	2.6422
23 Sep 2011 22:34:44	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	725,760	1,917,217	2.6417
22 Sep 2011 22:07:02	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	699,840	1,849,898	2.6433
21 Sep 2011 20:29:13	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	673,920	1,780,908	2.6426
20 Sep 2011 23:45:31	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	648,000	1,710,707	2.6400
19 Sep 2011 14:42:16	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	622,080	1,641,489	2.6387
18 Sep 2011 19:42:10	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	596,160	1,574,859	2.6417
18 Sep 2011 00:35:20	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	570,240	1,508,148	2.6448
17 Sep 2011 05:38:48	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	544,320	1,441,779	2.6488
16 Sep 2011 09:44:22	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	518,400	1,375,325	2.6530
15 Sep 2011 14:50:34	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	492,480	1,309,146	2.6583
14 Sep 2011 19:54:52	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	466,560	1,242,774	2.6637
14 Sep 2011 00:43:59	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	440,640	1,175,375	2.6674
13 Sep 2011 05:41:12	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	414,720	1,108,696	2.6734
11 Sep 2011 16:58:24	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	388,800	1,042,380	2.6810
10 Sep 2011 13:38:42	1006750	13289168	hadcm3n_p42s_1940_40_007420625_1	362,880	972,887	2.6810