Task 12736687

Name	hadcm3n_o1v8_1900_40_007197751_0
Workunit	7396031
Created	28 Mar 2011, 14:01:41 UTC
Sent	1 Apr 2011, 11:27:35 UTC
Report deadline	1 Jul 2011, 18:54:46 UTC
Received	25 May 2011, 4:11:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125685
Run time	36 days 2 hours 14 min 57 sec
CPU time	22 days 16 hours 57 min 34 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	1.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor -No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=768, selfPID=768, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 forrtl: There is not enough space on the disk. Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19924, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 May 2011 00:51:21	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	414,720	2,286,164	5.5125
17 May 2011 07:13:47	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	388,800	2,142,822	5.5114
15 May 2011 13:19:56	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	362,880	1,999,624	5.5104
13 May 2011 14:07:38	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	336,960	1,856,740	5.5103
26 Apr 2011 05:20:28	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	311,040	1,709,721	5.4968
24 Apr 2011 06:24:50	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	285,120	1,560,823	5.4743
20 Apr 2011 18:09:28	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	259,200	1,412,871	5.4509
20 Apr 2011 18:09:27	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	233,280	1,270,527	5.4464
20 Apr 2011 18:09:26	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	207,360	1,129,473	5.4469
20 Apr 2011 18:09:26	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	181,440	988,450	5.4478
12 Apr 2011 09:20:14	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	155,520	846,968	5.4460
10 Apr 2011 14:43:54	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	129,600	705,630	5.4447
08 Apr 2011 20:21:27	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	103,680	564,607	5.4457
07 Apr 2011 01:59:53	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	77,760	423,507	5.4463
05 Apr 2011 08:01:50	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	51,840	282,729	5.4539
03 Apr 2011 13:38:48	1125685	12736687	hadcm3n_o1v8_1900_40_007197751_0	25,920	141,234	5.4488