Task 15358478

Name	hadcm3n_o5h2_2100_40_008203847_2
Workunit	8358971
Created	9 Oct 2012, 15:46:21 UTC
Sent	9 Oct 2012, 15:46:36 UTC
Report deadline	8 Jan 2013, 23:13:47 UTC
Received	15 Dec 2012, 13:04:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1094475
Run time	9 days 18 hours 19 min 4 sec
CPU time	9 days 1 hours 10 min 24 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	3.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3356, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 03:31:16 (3432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. 05:02:21 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Dec 2012 09:30:45	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	492,480	761,585	1.5464
01 Dec 2012 19:38:16	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	466,560	720,061	1.5433
28 Nov 2012 20:18:25	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	440,640	680,683	1.5448
25 Nov 2012 15:56:54	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	414,720	641,104	1.5459
25 Nov 2012 02:44:09	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	388,800	603,967	1.5534
22 Nov 2012 16:30:56	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	362,880	565,310	1.5578
14 Nov 2012 23:38:09	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	336,960	528,205	1.5676
12 Nov 2012 18:37:44	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	311,040	489,312	1.5731
10 Nov 2012 19:54:23	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	285,120	450,006	1.5783
08 Nov 2012 18:45:23	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	259,200	410,024	1.5819
08 Nov 2012 02:18:38	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	233,280	372,959	1.5988
06 Nov 2012 17:47:15	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	207,360	334,260	1.6120
03 Nov 2012 13:40:27	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	181,440	283,541	1.5627
30 Oct 2012 10:25:11	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	155,520	239,750	1.5416
27 Oct 2012 15:22:02	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	129,600	198,646	1.5328
23 Oct 2012 21:44:31	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	103,680	160,812	1.5510
21 Oct 2012 08:34:55	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	77,760	122,538	1.5758
15 Oct 2012 19:50:43	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	51,840	83,175	1.6045
11 Oct 2012 19:46:31	1094475	15358478	hadcm3n_o5h2_2100_40_008203847_2	25,920	44,206	1.7055