Task 13925944

Name	hadcm3n_yeq3_1940_40_007683124_0
Workunit	7838211
Created	16 Jan 2012, 3:45:49 UTC
Sent	16 Jan 2012, 3:49:24 UTC
Report deadline	16 Apr 2012, 11:16:35 UTC
Received	2 Feb 2012, 6:46:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1185250
Run time	9 days 12 hours 20 min 3 sec
CPU time	8 days 3 hours 28 min 51 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:09:19 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:09:20 (5376): No heartbeat from core client for 30 sec - exiting 13:09:21 (5376): No heartbeat from core client for 30 sec - exiting 13:59:52 (6044): Can't acquire lockfile (32) - waiting 35s 14:00:05 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:03:13 (4316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:57:02 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: Not enough storage is available to process this command. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4780, iMonCtr=1 Model crash detected, will try to restart... 13:23:45 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: Not enough storage is available to process this command. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3144, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... cpdnmonitor: error reading file dataout/ocean_restart.day forrtl: Not enough storage is available to process this command. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:35 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:36 (4732): No heartbeat from core client for 30 sec - exiting 09:01:37 (4732): No heartbeat from core client for 30 sec - exiting 09:07:30 (4268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:34:06 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:26 (2712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:28 (4380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:02:46 (3164): Can't acquire lockfile (32) - waiting 35s 10:03:06 (5676): Can't acquire lockfile (32) - waiting 35s 10:03:11 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:03:41 (5676): Can't acquire lockfile (32) - exiting 10:03:41 (5676): Error: The process cannot access the file because it is being used by another process. (0x20) 10:04:12 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:43:42 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:18:20 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:30:28 (4436): Can't acquire lockfile (32) - waiting 35s 12:30:45 (3248): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: Not enough storage is available to process this command. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: Not enough storage is available to process this command. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Feb 2012 12:29:01	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	492,480	703,769	1.4290
31 Jan 2012 07:56:20	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	466,560	663,621	1.4224
30 Jan 2012 10:14:44	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	440,640	624,233	1.4167
27 Jan 2012 13:23:35	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	414,720	599,734	1.4461
25 Jan 2012 14:26:56	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	388,800	580,507	1.4931
25 Jan 2012 03:15:37	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	362,880	540,716	1.4901
24 Jan 2012 16:21:35	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	336,960	501,440	1.4881
24 Jan 2012 05:09:53	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	311,040	461,293	1.4831
23 Jan 2012 18:14:57	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	285,120	422,125	1.4805
23 Jan 2012 06:54:47	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	259,200	381,290	1.4710
22 Jan 2012 20:17:01	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	233,280	343,314	1.4717
22 Jan 2012 09:56:21	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	207,360	306,067	1.4760
21 Jan 2012 23:44:04	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	181,440	269,233	1.4839
21 Jan 2012 13:31:49	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	155,520	232,582	1.4955
21 Jan 2012 02:39:10	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	129,600	193,582	1.4937
20 Jan 2012 15:51:30	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	103,680	154,900	1.4940
20 Jan 2012 03:03:30	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	77,760	116,096	1.4930
19 Jan 2012 14:23:05	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	51,840	76,982	1.4850
18 Jan 2012 09:59:26	1185250	13925944	hadcm3n_yeq3_1940_40_007683124_0	25,920	38,536	1.4867