Task 15279390

Name	hadcm3n_zg71_1880_40_008200395_0
Workunit	8355519
Created	13 Sep 2012, 6:51:24 UTC
Sent	14 Sep 2012, 3:44:20 UTC
Report deadline	14 Dec 2012, 11:11:31 UTC
Received	11 Oct 2012, 14:09:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	775427
Run time	10 days 18 hours 3 min 50 sec
CPU time	9 days 23 hours 41 min 32 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.32 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 22:14:12 (12024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3452, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3488, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 14:50:19 (4056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:50:55 (1860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:50 (3328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:52 (3328): No heartbeat from core client for 30 sec - exiting 17:24:29 (6624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:48 (7264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:46 (4296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:47 (4296): No heartbeat from core client for 30 sec - exiting 08:01:48 (4296): No heartbeat from core client for 30 sec - exiting 08:01:49 (4296): No heartbeat from core client for 30 sec - exiting 08:01:50 (4296): No heartbeat from core client for 30 sec - exiting 08:01:51 (4296): No heartbeat from core client for 30 sec - exiting 08:01:52 (4296): No heartbeat from core client for 30 sec - exiting 08:01:53 (4296): No heartbeat from core client for 30 sec - exiting 14:26:21 (7932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:21:38 (4860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... C11:22:50 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:25:30 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:34:11 (9308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:20 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:36:28 (7396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:33:48 (3968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:11:22 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:48 (10084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9532, iMonCtr=1 Model crash detected, will try to restart... 19:44:44 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:45 (4480): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 20:46:06 (8312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5308, iMonCtr=1 Model crash detected, will try to restart... 22:47:51 (3284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:37 (6400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:04 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:03:39 (1256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:30 (3880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6788, iMonCtr=1 Model crash detected, will try to restart... 08:14:33 (612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:12:30 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:56:17 (8092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:40 (7508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4580, iMonCtr=1 Model crash detected, will try to restart... 13:23:06 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:24:35 (1400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:47:52 (7236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:27:08 (5188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7044, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Oct 2012 04:29:52	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	440,640	862,858	1.9582
10 Oct 2012 02:35:01	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	414,720	811,228	1.9561
09 Oct 2012 02:04:07	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	388,800	759,630	1.9538
30 Sep 2012 14:47:39	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	362,880	708,504	1.9524
29 Sep 2012 15:22:15	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	336,960	658,651	1.9547
28 Sep 2012 17:39:25	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	311,040	609,473	1.9595
27 Sep 2012 17:25:47	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	285,120	559,159	1.9611
26 Sep 2012 18:35:28	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	259,200	507,575	1.9582
25 Sep 2012 17:32:12	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	233,280	455,815	1.9539
24 Sep 2012 02:58:37	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	207,360	402,643	1.9418
23 Sep 2012 02:54:52	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	181,440	351,035	1.9347
22 Sep 2012 06:02:24	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	155,520	299,197	1.9238
21 Sep 2012 15:09:16	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	129,600	250,345	1.9317
20 Sep 2012 16:39:13	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	103,680	202,570	1.9538
18 Sep 2012 14:21:42	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	77,760	150,573	1.9364
17 Sep 2012 14:11:29	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	51,840	101,985	1.9673
16 Sep 2012 15:51:14	775427	15279390	hadcm3n_zg71_1880_40_008200395_0	25,920	52,173	2.0128