Task 15276592

Name	hadcm3n_zlkv_1880_40_008199436_1
Workunit	8354560
Created	13 Sep 2012, 0:52:46 UTC
Sent	13 Sep 2012, 0:56:50 UTC
Report deadline	13 Dec 2012, 8:24:01 UTC
Received	1 Oct 2012, 17:33:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1227761
Run time	9 days 6 hours 46 min 45 sec
CPU time	9 days 2 hours 40 min 17 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	3.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:21:20 (4592): No heartbeat from core client for 30 sec - exiting 17:21:22 (4592): No heartbeat from core client for 30 sec - exiting 17:21:23 (4592): No heartbeat from core client for 30 sec - exiting 17:21:24 (4592): No heartbeat from core client for 30 sec - exiting 17:21:25 (4592): No heartbeat from core client for 30 sec - exiting 17:21:26 (4592): No heartbeat from core client for 30 sec - exiting 17:21:27 (4592): No heartbeat from core client for 30 sec - exiting 17:21:28 (4592): No heartbeat from core client for 30 sec - exiting 17:21:29 (4592): No heartbeat from core client for 30 sec - exiting 17:21:30 (4592): No heartbeat from core client for 30 sec - exiting 17:21:31 (4592): No heartbeat from core client for 30 sec - exiting 17:21:32 (4592): No heartbeat from core client for 30 sec - exiting 17:21:34 (4592): No heartbeat from core client for 30 sec - exiting 17:21:35 (4592): No heartbeat from core client for 30 sec - exiting 17:21:36 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:15:23 (7928): Can't acquire lockfile (32) - waiting 35s 21:15:51 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7928, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1296, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:50:15 (3696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:03 (7380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1568, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2012 13:47:28	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	673,920	764,006	1.1337
30 Sep 2012 05:35:58	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	648,000	734,956	1.1342
29 Sep 2012 22:24:10	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	622,080	706,886	1.1363
29 Sep 2012 14:06:55	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	596,160	679,939	1.1405
28 Sep 2012 16:23:10	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	570,240	650,554	1.1408
27 Sep 2012 22:28:50	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	544,320	620,934	1.1408
26 Sep 2012 16:42:00	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	518,400	590,885	1.1398
25 Sep 2012 22:14:52	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	492,480	560,829	1.1388
25 Sep 2012 03:06:46	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	466,560	530,662	1.1374
24 Sep 2012 05:44:13	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	440,640	500,465	1.1358
23 Sep 2012 22:07:10	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	414,720	470,818	1.1353
23 Sep 2012 12:52:05	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	388,800	441,144	1.1346
23 Sep 2012 01:29:32	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	362,880	412,663	1.1372
22 Sep 2012 16:29:46	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	336,960	383,664	1.1386
22 Sep 2012 08:37:55	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	311,040	354,842	1.1408
22 Sep 2012 00:21:03	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	285,120	326,032	1.1435
21 Sep 2012 05:52:05	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	259,200	296,512	1.1440
20 Sep 2012 21:19:12	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	233,280	267,092	1.1449
19 Sep 2012 01:51:21	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	207,360	236,954	1.1427
18 Sep 2012 18:02:12	1227761	15276592	hadcm3n_zlkv_1880_40_008199436_1	181,440	207,123	1.1416