Task 15813748

Name	hadcm3n_o0qt_1940_40_008382272_0
Workunit	8533131
Created	1 Jun 2013, 4:15:30 UTC
Sent	1 Jun 2013, 4:20:14 UTC
Report deadline	31 Aug 2013, 11:47:25 UTC
Received	19 Jun 2013, 10:11:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1256704
Run time	1 days 12 hours 14 min 36 sec
CPU time	1 days 8 hours 49 min 30 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	1.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:37:10 (4824): No heartbeat from core client for 30 sec - exiting 10:37:11 (4824): No heartbeat from core client for 30 sec - exiting 10:37:12 (4824): No heartbeat from core client for 30 sec - exiting 10:37:13 (4824): No heartbeat from core client for 30 sec - exiting 10:37:14 (4824): No heartbeat from core client for 30 sec - exiting 10:37:15 (4824): No heartbeat from core client for 30 sec - exiting 10:37:16 (4824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2392, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 15:25:21 (6256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:42:03 (5484): No heartbeat from core client for 30 sec - exiting 17:42:04 (5484): No heartbeat from core client for 30 sec - exiting 17:42:05 (5484): No heartbeat from core client for 30 sec - exiting 17:42:06 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4744, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:43:29 (6056): No heartbeat from core client for 30 sec - exiting 09:43:30 (6056): No heartbeat from core client for 30 sec - exiting 09:43:31 (6056): No heartbeat from core client for 30 sec - exiting 09:43:32 (6056): No heartbeat from core client for 30 sec - exiting 09:43:33 (6056): No heartbeat from core client for 30 sec - exiting 09:43:34 (6056): No heartbeat from core client for 30 sec - exiting 09:43:35 (6056): No heartbeat from core client for 30 sec - exiting 09:43:36 (6056): No heartbeat from core client for 30 sec - exiting 09:43:37 (6056): No heartbeat from core client for 30 sec - exiting 09:43:39 (6056): No heartbeat from core client for 30 sec - exiting 09:43:40 (6056): No heartbeat from core client for 30 sec - exiting 09:43:41 (6056): No heartbeat from core client for 30 sec - exiting 09:43:42 (6056): No heartbeat from core client for 30 sec - exiting 09:43:43 (6056): No heartbeat from core client for 30 sec - exiting 09:43:44 (6056): No heartbeat from core client for 30 sec - exiting 09:43:45 (6056): No heartbeat from core client for 30 sec - exiting 09:43:46 (6056): No heartbeat from core client for 30 sec - exiting 09:43:47 (6056): No heartbeat from core client for 30 sec - exiting 09:43:48 (6056): No heartbeat from core client for 30 sec - exiting 09:43:50 (6056): No heartbeat from core client for 30 sec - exiting 09:43:51 (6056): No heartbeat from core client for 30 sec - exiting 09:43:52 (6056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:38:59 (4728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:33 (6196): No heartbeat from core client for 30 sec - exiting 09:23:34 (6196): No heartbeat from core client for 30 sec - exiting 09:23:35 (6196): No heartbeat from core client for 30 sec - exiting 09:23:36 (6196): No heartbeat from core client for 30 sec - exiting 09:23:37 (6196): No heartbeat from core client for 30 sec - exiting 09:23:38 (6196): No heartbeat from core client for 30 sec - exiting 09:23:39 (6196): No heartbeat from core client for 30 sec - exiting 09:23:40 (6196): No heartbeat from core client for 30 sec - exiting 09:23:41 (6196): No heartbeat from core client for 30 sec - exiting 09:23:42 (6196): No heartbeat from core client for 30 sec - exiting 09:23:43 (6196): No heartbeat from core client for 30 sec - exiting 09:23:44 (6196): No heartbeat from core client for 30 sec - exiting 09:23:45 (6196): No heartbeat from core client for 30 sec - exiting 09:23:46 (6196): No heartbeat from core client for 30 sec - exiting 09:23:47 (6196): No heartbeat from core client for 30 sec - exiting 09:23:48 (6196): No heartbeat from core client for 30 sec - exiting 09:23:49 (6196): No heartbeat from core client for 30 sec - exiting 09:23:50 (6196): No heartbeat from core client for 30 sec - exiting 09:23:51 (6196): No heartbeat from core client for 30 sec - exiting 09:23:52 (6196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:53 (6196): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:47:59 (6660): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:22:30 (4772): No heartbeat from core client for 30 sec - exiting 09:22:31 (4772): No heartbeat from core client for 30 sec - exiting 09:22:32 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:04:07 (6208): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4940, iMonCtr=1 Model crash detected, will try to restart... 17:36:04 (6156): No heartbeat from core client for 30 sec - exiting 17:36:05 (6156): No heartbeat from core client for 30 sec - exiting 17:36:06 (6156): No heartbeat from core client for 30 sec - exiting 17:36:07 (6156): No heartbeat from core client for 30 sec - exiting 17:36:08 (6156): No heartbeat from core client for 30 sec - exiting 17:36:09 (6156): No heartbeat from core client for 30 sec - exiting 17:36:10 (6156): No heartbeat from core client for 30 sec - exiting 17:36:11 (6156): No heartbeat from core client for 30 sec - exiting 17:36:12 (6156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:43:34 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:18:47 (7048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:20:59 (5812): No heartbeat from core client for 30 sec - exiting 18:21:00 (5812): No heartbeat from core client for 30 sec - exiting 18:21:01 (5812): No heartbeat from core client for 30 sec - exiting 18:21:02 (5812): No heartbeat from core client for 30 sec - exiting 18:21:03 (5812): No heartbeat from core client for 30 sec - exiting 18:21:04 (5812): No heartbeat from core client for 30 sec - exiting 18:21:05 (5812): No heartbeat from core client for 30 sec - exiting 18:21:06 (5812): No heartbeat from core client for 30 sec - exiting 18:21:07 (5812): No heartbeat from core client for 30 sec - exiting 18:21:08 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:38:00 (6120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:42:36 (5216): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 13:46:25 (6276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:57:50 (6180): No heartbeat from core client for 30 sec - exiting 08:57:51 (6180): No heartbeat from core client for 30 sec - exiting 08:57:52 (6180): No heartbeat from core client for 30 sec - exiting 08:57:53 (6180): No heartbeat from core client for 30 sec - exiting 08:57:54 (6180): No heartbeat from core client for 30 sec - exiting 08:57:55 (6180): No heartbeat from core client for 30 sec - exiting 08:57:56 (6180): No heartbeat from core client for 30 sec - exiting 08:57:57 (6180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:24:52 (6228): No heartbeat from core client for 30 sec - exiting 17:24:53 (6228): No heartbeat from core client for 30 sec - exiting 17:24:54 (6228): No heartbeat from core client for 30 sec - exiting 17:24:55 (6228): No heartbeat from core client for 30 sec - exiting 17:24:56 (6228): No heartbeat from core client for 30 sec - exiting 17:24:57 (6228): No heartbeat from core client for 30 sec - exiting 17:24:58 (6228): No heartbeat from core client for 30 sec - exiting 17:24:59 (6228): No heartbeat from core client for 30 sec - exiting 17:25:00 (6228): No heartbeat from core client for 30 sec - exiting 17:25:01 (6228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Jun 2013 00:00:50	1256704	15813748	hadcm3n_o0qt_1940_40_008382272_0	25,920	66,204	2.5542