Task 13390536

Name	hadcm3n_o43t_1940_40_007447698_3
Workunit	7645201
Created	16 Sep 2011, 1:02:15 UTC
Sent	16 Sep 2011, 1:06:39 UTC
Report deadline	16 Dec 2011, 8:33:50 UTC
Received	11 Dec 2011, 0:45:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	830038
Run time	21 days 11 hours 8 min 21 sec
CPU time	20 days 2 hours 59 min 52 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31376, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:31:32 (10696): No heartbeat from core client for 30 sec - exiting 03:31:33 (10696): No heartbeat from core client for 30 sec - exiting 03:31:34 (10696): No heartbeat from core client for 30 sec - exiting 03:31:35 (10696): No heartbeat from core client for 30 sec - exiting 03:31:36 (10696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:34:43 (10964): No heartbeat from core client for 30 sec - exiting 02:34:44 (10964): No heartbeat from core client for 30 sec - exiting 02:34:45 (10964): No heartbeat from core client for 30 sec - exiting 02:34:46 (10964): No heartbeat from core client for 30 sec - exiting 02:34:47 (10964): No heartbeat from core client for 30 sec - exiting 02:34:48 (10964): No heartbeat from core client for 30 sec - exiting 02:34:49 (10964): No heartbeat from core client for 30 sec - exiting 02:34:50 (10964): No heartbeat from core client for 30 sec - exiting 02:34:51 (10964): No heartbeat from core client for 30 sec - exiting 02:34:52 (10964): No heartbeat from core client for 30 sec - exiting 02:34:53 (10964): No heartbeat from core client for 30 sec - exiting 02:34:54 (10964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:41:28 (6168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:48:39 (10060): No heartbeat from core client for 30 sec - exiting 22:48:40 (10060): No heartbeat from core client for 30 sec - exiting 22:48:41 (10060): No heartbeat from core client for 30 sec - exiting 22:48:42 (10060): No heartbeat from core client for 30 sec - exiting 22:48:43 (10060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:34:28 (1704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:28:37 (7896): No heartbeat from core client for 30 sec - exiting 20:28:38 (7896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:05:36 (8164): No heartbeat from core client for 30 sec - exiting 04:05:37 (8164): No heartbeat from core client for 30 sec - exiting 04:05:38 (8164): No heartbeat from core client for 30 sec - exiting 04:05:39 (8164): No heartbeat from core client for 30 sec - exiting 04:05:40 (8164): No heartbeat from core client for 30 sec - exiting 04:05:41 (8164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9584, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Dec 2011 05:06:45	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	907,200	1,719,795	1.8957
07 Dec 2011 09:11:54	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	881,280	1,664,440	1.8887
05 Dec 2011 06:10:18	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	855,360	1,612,259	1.8849
15 Nov 2011 16:59:16	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	829,440	1,556,202	1.8762
15 Nov 2011 16:59:16	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	803,520	1,506,604	1.8750
10 Nov 2011 05:34:08	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	777,600	1,457,895	1.8749
08 Nov 2011 02:59:34	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	751,680	1,407,887	1.8730
05 Nov 2011 06:41:48	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	725,760	1,360,040	1.8740
03 Nov 2011 08:07:49	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	699,840	1,306,495	1.8668
17 Oct 2011 07:34:53	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	673,920	1,250,803	1.8560
15 Oct 2011 05:54:50	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	648,000	1,201,245	1.8538
14 Oct 2011 14:14:31	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	622,080	1,151,814	1.8516
14 Oct 2011 00:52:13	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	596,160	1,106,212	1.8556
12 Oct 2011 06:23:04	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	570,240	1,059,121	1.8573
11 Oct 2011 03:43:53	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	544,320	1,006,823	1.8497
09 Oct 2011 04:33:01	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	518,400	961,093	1.8540
07 Oct 2011 08:15:18	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	492,480	913,906	1.8557
06 Oct 2011 18:38:02	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	466,560	867,768	1.8599
06 Oct 2011 05:09:34	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	440,640	820,911	1.8630
05 Oct 2011 14:31:00	830038	13390536	hadcm3n_o43t_1940_40_007447698_3	414,720	772,514	1.8627