Task 13348877

Name	hadcm3n_o3rl_1940_40_007443421_0
Workunit	7640924
Created	8 Sep 2011, 23:49:51 UTC
Sent	8 Sep 2011, 23:55:12 UTC
Report deadline	9 Dec 2011, 7:22:23 UTC
Received	21 Nov 2011, 7:23:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1149499
Run time	8 days 5 hours 11 min 21 sec
CPU time	6 days 12 hours 44 min 34 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	2.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5136, iMonCtr=1 Model crash detected, will try to restart... Ocean Restart file copy failed on o3rlko.dae3350 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:44:59 (2476): Can't acquire lockfile (32) - waiting 35s Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5732, selfPID=5732, iMonCtr=1 No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5552, selfPID=5552, iMonCtr=1 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:14:47 (5004): No heartbeat from core client for 30 sec - exiting 18:14:49 (5004): No heartbeat from core client for 30 sec - exiting 18:14:50 (5004): No heartbeat from core client for 30 sec - exiting 18:14:51 (5004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:52 (5004): No heartbeat from core client for 30 sec - exiting Ocean Restart file copy failed on o3rlko.dae92m0 Ocean Restart file copy failed on o3rlko.dae92n0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4912, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3rlko.dae9c60 Ocean Restart file copy failed on o3rlko.dae9c70 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on o3rlko.daf1co0 Ocean Restart file copy failed on o3rlko.daf2130 Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3rlko.daf31i0 Ocean Restart file copy failed on o3rlko.daf31j0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on o3rlko.daf4690 Ocean Restart file copy failed on o3rlko.daf46a0 Ocean Restart file copy failed on o3rlko.daf46b0 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on o3rlko.daf64l0 Ocean Restart file copy failed on o3rlko.daf64m0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5760, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3rlko.daf7770 Ocean Restart file copy failed on o3rlko.daf7780 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on o3rlko.daf78e0 Ocean Restart file copy failed on o3rlko.daf78f0 CPDN Monitor - Quit request from BOINC... Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Nov 2011 11:04:47	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	414,720	540,975	1.3044
17 Nov 2011 22:46:18	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	388,800	507,531	1.3054
17 Nov 2011 10:58:57	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	362,880	474,423	1.3074
17 Nov 2011 00:06:09	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	336,960	441,412	1.3100
16 Nov 2011 02:06:54	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	311,040	407,642	1.3106
15 Nov 2011 20:07:44	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	285,120	374,380	1.3131
15 Nov 2011 20:07:44	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	259,200	341,223	1.3164
15 Nov 2011 20:07:44	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	233,280	308,700	1.3233
06 Nov 2011 01:35:11	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	207,360	275,696	1.3296
04 Nov 2011 21:19:22	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	181,440	242,575	1.3369
06 Oct 2011 21:58:31	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	155,520	208,211	1.3388
05 Oct 2011 17:24:52	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	129,600	173,992	1.3425
03 Oct 2011 15:41:34	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	103,680	138,849	1.3392
02 Oct 2011 18:38:20	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	77,760	103,807	1.3350
27 Sep 2011 00:02:17	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	51,840	69,111	1.3332
26 Sep 2011 13:02:38	1149499	13348877	hadcm3n_o3rl_1940_40_007443421_0	25,920	34,369	1.3260