Task 15812081

Name	hadcm3n_o7co_1940_40_008380914_0
Workunit	8531773
Created	31 May 2013, 23:29:57 UTC
Sent	15 Jun 2013, 23:33:32 UTC
Report deadline	15 Sep 2013, 7:00:43 UTC
Received	5 Jul 2013, 18:53:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1255904
Run time	16 days 16 hours 8 min 41 sec
CPU time	7 days 12 hours 35 min 11 sec
Validate state	Invalid
Credit	10,575.36
Device peak FLOPS	2.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> cpdnmonitor: error reading file dataout/ocean_restart.day 21:40:05 (8468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8956, iMonCtr=1 Model crash detected, will try to restart... forrtl: Not enough storage is available to process this command. 03:00:33 (8956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:34:14 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:15:49 (4048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:30 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 02:56:53 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7184, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7184, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7184, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7184, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Jul 2013 05:16:13	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	881,280	672,654	0.7633
06 Jul 2013 04:34:44	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	855,360	640,306	0.7486
04 Jul 2013 15:50:25	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	829,440	1,234,349	1.4882
04 Jul 2013 14:11:57	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	803,520	1,192,885	1.4846
03 Jul 2013 15:13:40	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	777,600	1,150,646	1.4797
03 Jul 2013 03:24:09	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	751,680	1,108,841	1.4752
02 Jul 2013 15:40:58	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	725,760	1,067,313	1.4706
02 Jul 2013 12:03:58	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	699,840	1,025,773	1.4657
02 Jul 2013 11:49:42	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	673,920	984,016	1.4601
02 Jul 2013 11:14:34	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	648,000	942,467	1.4544
02 Jul 2013 10:50:07	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	622,080	901,348	1.4489
02 Jul 2013 10:30:09	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	596,160	860,002	1.4426
02 Jul 2013 10:13:18	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	570,240	817,688	1.4339
02 Jul 2013 09:58:11	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	544,320	775,929	1.4255
02 Jul 2013 09:45:20	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	518,400	734,799	1.4174
28 Jun 2013 05:42:11	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	492,480	693,001	1.4072
27 Jun 2013 18:03:48	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	466,560	651,661	1.3967
27 Jun 2013 06:12:53	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	440,640	700,657	1.5901
26 Jun 2013 18:33:27	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	414,720	658,781	1.5885
26 Jun 2013 06:18:31	1255904	15812081	hadcm3n_o7co_1940_40_008380914_0	388,800	616,402	1.5854