Task 15493886

Name	hadcm3n_3c38_1940_40_008264848_0
Workunit	8419972
Created	21 Dec 2012, 10:23:16 UTC
Sent	21 Dec 2012, 10:24:40 UTC
Report deadline	22 Mar 2013, 17:51:51 UTC
Received	14 Jan 2013, 9:29:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153594
Run time	8 days 14 hours 32 min 50 sec
CPU time	8 days 8 hours 38 min 58 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:51:30 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:06 (1088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... 09:57:04 (1908): No heartbeat from core client for 30 sec - exiting 09:57:05 (1908): No heartbeat from core client for 30 sec - exiting 09:57:06 (1908): No heartbeat from core client for 30 sec - exiting 09:57:07 (1908): No heartbeat from core client for 30 sec - exiting 09:57:08 (1908): No heartbeat from core client for 30 sec - exiting 09:57:09 (1908): No heartbeat from core client for 30 sec - exiting 09:57:11 (1908): No heartbeat from core client for 30 sec - exiting 09:57:12 (1908): No heartbeat from core client for 30 sec - exiting 09:57:13 (1908): No heartbeat from core client for 30 sec - exiting 09:57:14 (1908): No heartbeat from core client for 30 sec - exiting 09:57:15 (1908): No heartbeat from core client for 30 sec - exiting 09:57:16 (1908): No heartbeat from core client for 30 sec - exiting 09:57:17 (1908): No heartbeat from core client for 30 sec - exiting 09:57:18 (1908): No heartbeat from core client for 30 sec - exiting 09:57:19 (1908): No heartbeat from core client for 30 sec - exiting 09:57:20 (1908): No heartbeat from core client for 30 sec - exiting 09:57:21 (1908): No heartbeat from core client for 30 sec - exiting 09:57:23 (1908): No heartbeat from core client for 30 sec - exiting 09:57:24 (1908): No heartbeat from core client for 30 sec - exiting 09:57:25 (1908): No heartbeat from core client for 30 sec - exiting 09:57:26 (1908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3372, iMonCtr=1 Model crash detected, will try to restart... 15:14:06 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:52 (1120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:57:54 (4936): No heartbeat from core client for 30 sec - exiting 09:57:55 (4936): No heartbeat from core client for 30 sec - exiting 09:57:56 (4936): No heartbeat from core client for 30 sec - exiting 09:57:57 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:58 (4936): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jan 2013 06:18:53	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	570,240	720,655	1.2638
10 Jan 2013 21:16:33	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	544,320	688,786	1.2654
10 Jan 2013 11:19:06	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	518,400	656,556	1.2665
10 Jan 2013 02:02:30	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	492,480	623,892	1.2668
09 Jan 2013 17:26:08	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	466,560	591,605	1.2680
09 Jan 2013 08:00:04	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	440,640	559,409	1.2695
08 Jan 2013 22:02:34	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	414,720	526,918	1.2705
08 Jan 2013 12:50:14	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	388,800	494,538	1.2720
08 Jan 2013 03:20:51	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	362,880	461,158	1.2708
07 Jan 2013 17:47:40	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	336,960	427,593	1.2690
07 Jan 2013 08:20:52	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	311,040	394,239	1.2675
06 Jan 2013 22:53:40	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	285,120	361,103	1.2665
06 Jan 2013 13:22:03	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	259,200	327,630	1.2640
06 Jan 2013 03:55:03	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	233,280	294,590	1.2628
05 Jan 2013 18:41:26	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	207,360	262,003	1.2635
05 Jan 2013 09:25:57	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	181,440	229,820	1.2666
05 Jan 2013 01:04:01	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	155,520	197,419	1.2694
04 Jan 2013 15:51:06	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	129,600	165,236	1.2750
23 Dec 2012 00:26:18	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	103,680	132,854	1.2814
22 Dec 2012 14:50:39	1153594	15493886	hadcm3n_3c38_1940_40_008264848_0	77,760	99,336	1.2775