Task 13783144

Name	hadcm3n_u2g4_1980_40_007616277_1
Workunit	7794407
Created	15 Dec 2011, 15:16:42 UTC
Sent	15 Dec 2011, 15:54:29 UTC
Report deadline	15 Mar 2012, 23:21:40 UTC
Received	24 May 2012, 1:33:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1085859
Run time	98 days 16 hours 15 min 4 sec
CPU time	67 days 19 hours 41 min 48 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	0.73 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:14:50 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:01 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:29:42 (5244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:41 (1512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:23 (808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:55 (1468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:52:19 (3236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:42 (5956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:37 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:00 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:04 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:26:33 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:41 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:44 (3932): No heartbeat from core client for 30 sec - exiting 02:29:45 (3932): No heartbeat from core client for 30 sec - exiting 02:29:46 (3932): No heartbeat from core client for 30 sec - exiting 02:29:47 (3932): No heartbeat from core client for 30 sec - exiting 02:29:48 (3932): No heartbeat from core client for 30 sec - exiting 02:29:49 (3932): No heartbeat from core client for 30 sec - exiting 14:27:40 (4956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:27:41 (4956): No heartbeat from core client for 30 sec - exiting 14:27:42 (4956): No heartbeat from core client for 30 sec - exiting 14:27:43 (4956): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 22:24:15 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:58:04 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:14 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:24:16 (3224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:14:44 (1624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:46 (1624): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5108, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:14:24 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5212, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 15:07:46 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:47 (5732): No heartbeat from core client for 30 sec - exiting 22:47:27 (4088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:50:55 (6100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:30 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:00 (6028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:52:37 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:01 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:02 (4752): No heartbeat from core client for 30 sec - exiting 16:53:03 (4752): No heartbeat from core client for 30 sec - exiting 16:53:04 (4752): No heartbeat from core client for 30 sec - exiting 16:53:05 (4752): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 05:15:24 (3740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 May 2012 06:23:50	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	881,280	5,859,744	6.6491
17 May 2012 09:25:12	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	855,360	5,529,142	6.4641
11 May 2012 10:29:35	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	829,440	5,203,099	6.2730
07 May 2012 18:59:01	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	803,520	4,999,732	6.2223
06 May 2012 07:31:22	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	777,600	4,857,328	6.2466
04 May 2012 02:07:40	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	751,680	4,705,453	6.2599
01 May 2012 04:16:32	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	725,760	4,552,602	6.2729
29 Apr 2012 01:27:19	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	699,840	4,405,147	6.2945
29 Apr 2012 01:27:19	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	673,920	4,228,124	6.2739
24 Apr 2012 01:36:32	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	648,000	4,083,383	6.3015
20 Apr 2012 21:31:48	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	622,080	3,903,217	6.2745
18 Apr 2012 00:27:16	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	596,160	3,708,373	6.2204
10 Apr 2012 18:47:45	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	570,240	3,544,993	6.2167
08 Apr 2012 22:33:11	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	544,320	3,407,293	6.2597
05 Apr 2012 02:18:58	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	518,400	3,218,923	6.2093
08 Mar 2012 15:19:54	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	492,480	2,999,472	6.0905
04 Mar 2012 15:05:20	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	466,560	2,832,473	6.0710
26 Feb 2012 14:39:59	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	440,640	2,663,066	6.0436
18 Feb 2012 09:00:37	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	414,720	2,484,961	5.9919
11 Feb 2012 10:00:08	1085859	13783144	hadcm3n_u2g4_1980_40_007616277_1	388,800	2,300,904	5.9180