Task 13093405

Name	hadcm3n_y96w_1900_40_007344770_0
Workunit	7542200
Created	6 Jul 2011, 13:26:43 UTC
Sent	22 Jul 2011, 13:12:07 UTC
Report deadline	21 Oct 2011, 20:39:18 UTC
Received	29 Aug 2011, 14:52:57 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1148484
Run time	29 days 16 hours 6 min 27 sec
CPU time	15 days 21 hours 24 min 58 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=63664, selfPID=63664, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 07:58:37 (18852): No heartbeat from core client for 30 sec - exiting 07:58:42 (18852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:45 (18852): No heartbeat from core client for 30 sec - exiting 08:03:14 (26572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:16 (26572): No heartbeat from core client for 30 sec - exiting 08:07:08 (21260): No heartbeat from core client for 30 sec - exiting 08:07:10 (21260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:15 (21260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:23:46 (43408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:48 (43408): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/y96wko.pja9c10 Error converting file to netcdf: dataout/y96wko.pia9c10 Error converting file to netcdf: dataout/y96wko.pfa9c10 Error converting file to netcdf: dataout/y96wka.pha9c10 Error converting file to netcdf: dataout/y96wka.pga9c10 Error converting file to netcdf: dataout/y96wka.pea9c10 Error converting file to netcdf: dataout/y96wka.pda9c10 CPDN Monitor - Quit request from BOINC... 17:55:10 (59732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:55:12 (59732): No heartbeat from core client for 30 sec - exiting 18:01:22 (60976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:01:24 (60976): No heartbeat from core client for 30 sec - exiting 18:05:39 (72104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:42 (72104): No heartbeat from core client for 30 sec - exiting 18:09:13 (72080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:14 (72080): No heartbeat from core client for 30 sec - exiting 18:13:07 (70560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:17 (70560): No heartbeat from core client for 30 sec - exiting 18:15:20 (70360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:22 (70360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:40:11 (6548): No heartbeat from core client for 30 sec - exiting 18:40:14 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:16 (6548): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9936, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:03:00 (5920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:06 (5920): No heartbeat from core client for 30 sec - exiting 03:29:25 (5348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Aug 2011 07:09:23	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	518,400	1,373,145	2.6488
26 Aug 2011 19:12:05	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	492,480	1,305,916	2.6517
22 Aug 2011 22:12:23	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	466,560	1,238,911	2.6554
21 Aug 2011 11:11:33	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	440,640	1,174,580	2.6656
20 Aug 2011 02:12:44	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	414,720	1,105,174	2.6649
18 Aug 2011 10:55:41	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	388,800	1,036,392	2.6656
15 Aug 2011 11:08:06	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	362,880	973,850	2.6837
13 Aug 2011 22:57:20	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	336,960	901,977	2.6768
11 Aug 2011 21:19:41	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	311,040	832,046	2.6750
09 Aug 2011 22:16:56	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	285,120	768,787	2.6964
07 Aug 2011 03:47:11	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	259,200	700,028	2.7007
04 Aug 2011 08:20:20	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	233,280	633,494	2.7156
02 Aug 2011 21:52:12	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	207,360	564,361	2.7216
01 Aug 2011 19:00:48	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	181,440	493,264	2.7186
31 Jul 2011 17:38:43	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	155,520	420,099	2.7013
30 Jul 2011 16:12:12	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	129,600	348,462	2.6888
28 Jul 2011 21:15:49	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	103,680	278,267	2.6839
27 Jul 2011 04:25:09	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	77,760	210,595	2.7083
25 Jul 2011 23:02:38	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	51,840	142,359	2.7461
25 Jul 2011 21:13:45	1148484	13093405	hadcm3n_y96w_1900_40_007344770_0	25,920	72,595	2.8007