Task 15475096

Name	hadcm3n_zfus_1880_40_008249709_2
Workunit	8404833
Created	13 Dec 2012, 18:35:28 UTC
Sent	13 Dec 2012, 18:40:28 UTC
Report deadline	15 Mar 2013, 2:07:39 UTC
Received	22 Dec 2012, 4:50:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1253823
Run time	6 days 9 hours 31 min 8 sec
CPU time	5 days 18 hours 0 min 28 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:05:05 (10144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:06 (10144): No heartbeat from core client for 30 sec - exiting 06:45:32 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:45:35 (5332): No heartbeat from core client for 30 sec - exiting 06:45:37 (5332): No heartbeat from core client for 30 sec - exiting 06:45:39 (5332): No heartbeat from core client for 30 sec - exiting 06:45:42 (5332): No heartbeat from core client for 30 sec - exiting 06:45:44 (5332): No heartbeat from core client for 30 sec - exiting 06:48:43 (8208): No heartbeat from core client for 30 sec - exiting 06:48:44 (8208): No heartbeat from core client for 30 sec - exiting 06:48:46 (8208): No heartbeat from core client for 30 sec - exiting 06:48:48 (8208): No heartbeat from core client for 30 sec - exiting 06:48:50 (8208): No heartbeat from core client for 30 sec - exiting 06:48:52 (8208): No heartbeat from core client for 30 sec - exiting 06:48:54 (8208): No heartbeat from core client for 30 sec - exiting 06:48:58 (8208): No heartbeat from core client for 30 sec - exiting 06:49:00 (8208): No heartbeat from core client for 30 sec - exiting 06:49:02 (8208): No heartbeat from core client for 30 sec - exiting 06:49:06 (8208): No heartbeat from core client for 30 sec - exiting 06:49:11 (8208): No heartbeat from core client for 30 sec - exiting 06:49:13 (8208): No heartbeat from core client for 30 sec - exiting 06:49:16 (8208): No heartbeat from core client for 30 sec - exiting 06:49:19 (8208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:16 (8708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:18 (8708): No heartbeat from core client for 30 sec - exiting 07:07:45 (4816): No heartbeat from core client for 30 sec - exiting 07:07:46 (4816): No heartbeat from core client for 30 sec - exiting 07:07:48 (4816): No heartbeat from core client for 30 sec - exiting 07:07:50 (4816): No heartbeat from core client for 30 sec - exiting 07:07:51 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:55 (2468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:31:58 (2468): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:02:22 (9108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:30:57 (6556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:00 (9360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:02 (9360): No heartbeat from core client for 30 sec - exiting 08:35:04 (9360): No heartbeat from core client for 30 sec - exiting 08:40:52 (7288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:53 (7288): No heartbeat from core client for 30 sec - exiting 08:45:20 (7248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:22 (7248): No heartbeat from core client for 30 sec - exiting 08:47:58 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:24 (7212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:25 (7212): No heartbeat from core client for 30 sec - exiting 09:13:36 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:37 (5052): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9240, iMonCtr=1 Model crash detected, will try to restart... 16:11:23 (4352): No heartbeat from core client for 30 sec - exiting 16:11:24 (4352): No heartbeat from core client for 30 sec - exiting 16:11:25 (4352): No heartbeat from core client for 30 sec - exiting 16:11:26 (4352): No heartbeat from core client for 30 sec - exiting 16:11:27 (4352): No heartbeat from core client for 30 sec - exiting 16:11:28 (4352): No heartbeat from core client for 30 sec - exiting 16:11:29 (4352): No heartbeat from core client for 30 sec - exiting 16:11:30 (4352): No heartbeat from core client for 30 sec - exiting 16:11:31 (4352): No heartbeat from core client for 30 sec - exiting 16:11:33 (4352): No heartbeat from core client for 30 sec - exiting 16:11:34 (4352): No heartbeat from core client for 30 sec - exiting 16:11:35 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:02:56 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:08 (6252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:01:39 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4140, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2012 17:29:13	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	207,360	488,282	2.3548
19 Dec 2012 00:05:17	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	181,440	426,673	2.3516
18 Dec 2012 05:31:06	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	155,520	364,638	2.3446
17 Dec 2012 10:55:05	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	129,600	302,114	2.3311
16 Dec 2012 17:42:52	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	103,680	241,782	2.3320
15 Dec 2012 22:42:28	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	77,760	182,092	2.3417
15 Dec 2012 05:32:56	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	51,840	121,835	2.3502
14 Dec 2012 12:20:20	1253823	15475096	hadcm3n_zfus_1880_40_008249709_2	25,920	60,927	2.3506