Task 13350849

Name	hadcm3n_o77x_1940_40_007444187_0
Workunit	7641690
Created	9 Sep 2011, 1:45:30 UTC
Sent	9 Sep 2011, 5:56:13 UTC
Report deadline	9 Dec 2011, 13:23:24 UTC
Received	18 Sep 2011, 3:15:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1050240
Run time	5 days 7 hours 17 min 1 sec
CPU time	3 days 14 hours 44 min 32 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30789, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30789, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30789, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:29:19 (8662): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... 06:24:56 (23855): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:35 (23885): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:38 (23885): No heartbeat from core client for 30 sec - exiting 07:07:39 (23885): No heartbeat from core client for 30 sec - exiting 10:04:38 (23915): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:03:16 (24226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:21 (24226): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26070, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Sep 2011 17:22:24	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	155,520	268,184	1.7244
13 Sep 2011 23:02:19	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	129,600	262,400	2.0247
13 Sep 2011 03:32:10	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	103,680	198,648	1.9160
12 Sep 2011 09:42:10	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	77,760	137,155	1.7638
11 Sep 2011 14:13:28	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	51,840	87,912	1.6958
10 Sep 2011 18:19:15	1050240	13350849	hadcm3n_o77x_1940_40_007444187_0	25,920	25,706	0.9917