Task 15461298

Name	hadcm3n_z9za_1880_40_008253473_1
Workunit	8408597
Created	26 Nov 2012, 10:42:25 UTC
Sent	26 Nov 2012, 10:42:35 UTC
Report deadline	25 Feb 2013, 18:09:46 UTC
Received	6 Dec 2012, 7:44:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	875724
Run time	8 days 7 hours 14 min 57 sec
CPU time	7 days 23 hours 55 min 28 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 11:33:23 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:26 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:29 (5716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:30 (5716): No heartbeat from core client for 30 sec - exiting 11:41:31 (5716): No heartbeat from core client for 30 sec - exiting 11:41:32 (5716): No heartbeat from core client for 30 sec - exiting 11:41:33 (5716): No heartbeat from core client for 30 sec - exiting 11:50:12 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:13 (5940): No heartbeat from core client for 30 sec - exiting 11:50:14 (5940): No heartbeat from core client for 30 sec - exiting 11:50:15 (5940): No heartbeat from core client for 30 sec - exiting 12:00:13 (3928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:17 (3928): No heartbeat from core client for 30 sec - exiting 12:00:18 (3928): No heartbeat from core client for 30 sec - exiting 12:00:19 (3928): No heartbeat from core client for 30 sec - exiting 12:00:20 (3928): No heartbeat from core client for 30 sec - exiting 12:00:21 (3928): No heartbeat from core client for 30 sec - exiting 12:00:22 (3928): No heartbeat from core client for 30 sec - exiting 12:05:13 (3524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:18 (2648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:10:23 (2648): No heartbeat from core client for 30 sec - exiting 12:10:24 (2648): No heartbeat from core client for 30 sec - exiting 12:17:17 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:17:23 (4924): No heartbeat from core client for 30 sec - exiting 12:17:24 (4924): No heartbeat from core client for 30 sec - exiting 12:17:25 (4924): No heartbeat from core client for 30 sec - exiting 12:17:26 (4924): No heartbeat from core client for 30 sec - exiting 12:17:27 (4924): No heartbeat from core client for 30 sec - exiting 12:17:28 (4924): No heartbeat from core client for 30 sec - exiting 12:17:29 (4924): No heartbeat from core client for 30 sec - exiting 12:17:30 (4924): No heartbeat from core client for 30 sec - exiting 12:17:31 (4924): No heartbeat from core client for 30 sec - exiting 12:17:32 (4924): No heartbeat from core client for 30 sec - exiting 12:17:33 (4924): No heartbeat from core client for 30 sec - exiting 12:17:34 (4924): No heartbeat from core client for 30 sec - exiting 12:17:35 (4924): No heartbeat from core client for 30 sec - exiting 12:17:36 (4924): No heartbeat from core client for 30 sec - exiting 12:20:17 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:20:18 (4952): No heartbeat from core client for 30 sec - exiting 12:20:19 (4952): No heartbeat from core client for 30 sec - exiting 12:25:18 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:19 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:43:31 (5220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:27:26 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:10:36 (2920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:33:20 (6740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:15:00 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:01:06 (6680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:00:41 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:03:02 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:58:14 (7640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:26:48 (7676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:46:16 (7812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:46:15 (9156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:10 (2504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:11 (2504): No heartbeat from core client for 30 sec - exiting 16:48:12 (2504): No heartbeat from core client for 30 sec - exiting 16:48:13 (2504): No heartbeat from core client for 30 sec - exiting 16:48:14 (2504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:46:12 (3792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:46:11 (1184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:08:50 (8900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:18:37 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:02:01 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:56:02 (3400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:28:57 (7904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:22:13 (8908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:40:11 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1232, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 22:42:47 (1232): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2624, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Dec 2012 01:01:54	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	596,160	686,103	1.1509
05 Dec 2012 12:55:24	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	570,240	655,672	1.1498
05 Dec 2012 03:38:23	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	544,320	625,662	1.1494
04 Dec 2012 19:04:05	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	518,400	595,191	1.1481
04 Dec 2012 10:07:31	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	492,480	563,663	1.1445
04 Dec 2012 01:30:35	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	466,560	533,039	1.1425
03 Dec 2012 17:26:45	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	440,640	504,605	1.1452
03 Dec 2012 09:14:26	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	414,720	475,481	1.1465
03 Dec 2012 00:37:47	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	388,800	446,024	1.1472
02 Dec 2012 15:54:57	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	362,880	415,727	1.1456
02 Dec 2012 07:53:27	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	336,960	387,136	1.1489
01 Dec 2012 22:41:31	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	311,040	357,195	1.1484
01 Dec 2012 14:17:31	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	285,120	326,608	1.1455
01 Dec 2012 06:00:38	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	259,200	298,412	1.1513
30 Nov 2012 22:03:47	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	233,280	270,261	1.1585
30 Nov 2012 13:08:43	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	207,360	240,007	1.1574
30 Nov 2012 04:20:51	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	181,440	210,260	1.1588
29 Nov 2012 19:42:13	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	155,520	180,538	1.1609
29 Nov 2012 10:47:57	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	129,600	152,155	1.1740
29 Nov 2012 00:35:45	875724	15461298	hadcm3n_z9za_1880_40_008253473_1	103,680	123,495	1.1911