Task 13215451

Name	hadcm3n_ydud_1900_40_007350799_2
Workunit	7548229
Created	8 Aug 2011, 10:43:44 UTC
Sent	8 Aug 2011, 10:43:52 UTC
Report deadline	7 Nov 2011, 18:11:03 UTC
Received	7 Feb 2012, 10:42:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	946571
Run time	31 days 6 hours 45 min 53 sec
CPU time	29 days 1 hours 4 min 58 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:41:59 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:00 (5232): No heartbeat from core client for 30 sec - exiting 16:42:01 (5232): No heartbeat from core client for 30 sec - exiting 16:42:02 (5232): No heartbeat from core client for 30 sec - exiting 16:42:03 (5232): No heartbeat from core client for 30 sec - exiting 16:42:04 (5232): No heartbeat from core client for 30 sec - exiting 16:42:06 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2824, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:00:15 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:18 (5020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:45:57 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:50 (1092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:51:02 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:09:50 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:45 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:37:21 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4812, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Feb 2012 09:09:43	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	907,200	2,484,080	2.7382
18 Jan 2012 11:09:42	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	881,280	2,434,194	2.7621
16 Jan 2012 13:41:19	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	855,360	2,384,790	2.7881
12 Jan 2012 16:06:15	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	829,440	2,335,048	2.8152
11 Jan 2012 10:14:17	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	803,520	2,285,115	2.8439
09 Jan 2012 12:49:21	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	777,600	2,235,509	2.8749
05 Jan 2012 14:20:53	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	751,680	2,185,830	2.9079
03 Jan 2012 11:36:53	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	725,760	2,135,170	2.9420
29 Dec 2011 08:53:58	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	699,840	2,085,479	2.9799
20 Dec 2011 11:30:00	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	673,920	2,035,897	3.0210
16 Dec 2011 13:13:00	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	648,000	1,986,388	3.0654
14 Dec 2011 15:55:06	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	622,080	1,936,877	3.1135
12 Dec 2011 16:46:05	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	596,160	1,887,112	3.1654
09 Dec 2011 11:03:14	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	570,240	1,837,698	3.2227
07 Dec 2011 10:12:10	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	544,320	1,788,265	3.2853
05 Dec 2011 12:01:48	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	518,400	1,738,616	3.3538
01 Dec 2011 13:52:44	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	492,480	1,689,062	3.4297
29 Nov 2011 17:08:28	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	466,560	1,639,246	3.5135
28 Nov 2011 11:06:16	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	440,640	1,602,437	3.6366
03 Oct 2011 12:11:16	946571	13215451	hadcm3n_ydud_1900_40_007350799_2	414,720	767,700	1.8511