Task 14032407

Name	hadcm3n_u2p3_1980_40_007742906_2
Workunit	7898014
Created	30 Jan 2012, 14:25:56 UTC
Sent	30 Jan 2012, 14:36:09 UTC
Report deadline	30 Apr 2012, 22:03:20 UTC
Received	13 Mar 2012, 16:44:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1193948
Run time	6 days 4 hours 22 min 10 sec
CPU time	5 days 19 hours 43 min 38 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... C17:50:11 (5820): No heartbeat from core client for 30 sec - exiting 17:50:12 (5820): No heartbeat from core client for 30 sec - exiting 17:50:13 (5820): No heartbeat from core client for 30 sec - exiting 17:50:14 (5820): No heartbeat from core client for 30 sec - exiting 17:50:15 (5820): No heartbeat from core client for 30 sec - exiting 17:50:16 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4952, selfPID=4952, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u2p3ko.pji5c10 Error converting file to netcdf: dataout/u2p3ko.pii5c10 Error converting file to netcdf: dataout/u2p3ko.pfi5c10 Error converting file to netcdf: dataout/u2p3ka.phi5c10 Error converting file to netcdf: dataout/u2p3ka.pgi5c10 Error converting file to netcdf: dataout/u2p3ka.pei5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:44:04 (2412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:27:16 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:42 (10528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:08:09 (6644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:05:24 (4088): No heartbeat from core client for 30 sec - exiting 09:05:25 (4088): No heartbeat from core client for 30 sec - exiting 09:05:26 (4088): No heartbeat from core client for 30 sec - exiting 09:05:27 (4088): No heartbeat from core client for 30 sec - exiting 09:05:28 (4088): No heartbeat from core client for 30 sec - exiting 09:05:30 (4088): No heartbeat from core client for 30 sec - exiting 09:05:31 (4088): No heartbeat from core client for 30 sec - exiting 09:05:32 (4088): No heartbeat from core client for 30 sec - exiting 09:05:33 (4088): No heartbeat from core client for 30 sec - exiting 09:05:34 (4088): No heartbeat from core client for 30 sec - exiting 09:05:35 (4088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:14:37 (5708): No heartbeat from core client for 30 sec - exiting 09:14:38 (5708): No heartbeat from core client for 30 sec - exiting 09:14:39 (5708): No heartbeat from core client for 30 sec - exiting 09:14:40 (5708): No heartbeat from core client for 30 sec - exiting 09:14:42 (5708): No heartbeat from core client for 30 sec - exiting 09:14:43 (5708): No heartbeat from core client for 30 sec - exiting 09:14:44 (5708): No heartbeat from core client for 30 sec - exiting 09:14:45 (5708): No heartbeat from core client for 30 sec - exiting 09:14:46 (5708): No heartbeat from core client for 30 sec - exiting 09:14:47 (5708): No heartbeat from core client for 30 sec - exiting 09:14:48 (5708): No heartbeat from core client for 30 sec - exiting 09:14:49 (5708): No heartbeat from core client for 30 sec - exiting 09:14:50 (5708): No heartbeat from core client for 30 sec - exiting 09:14:52 (5708): No heartbeat from core client for 30 sec - exiting 09:14:53 (5708): No heartbeat from core client for 30 sec - exiting 09:14:54 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:55 (5708): No heartbeat from core client for 30 sec - exiting 09:16:37 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10548, selfPID=10548, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7572, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 09:09:44 (5472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:20 (1080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:21:44 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Mar 2012 15:20:31	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	414,720	492,983	1.1887
08 Mar 2012 22:05:36	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	388,800	467,920	1.2035
07 Mar 2012 15:12:04	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	362,880	442,346	1.2190
05 Mar 2012 19:16:35	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	336,960	416,562	1.2362
01 Mar 2012 22:30:08	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	311,040	390,887	1.2567
01 Mar 2012 15:14:27	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	285,120	364,843	1.2796
28 Feb 2012 19:33:23	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	259,200	338,783	1.3070
23 Feb 2012 17:52:25	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	233,280	311,231	1.3342
22 Feb 2012 17:46:40	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	207,360	282,774	1.3637
17 Feb 2012 17:31:01	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	181,440	249,358	1.3743
14 Feb 2012 20:12:44	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	155,520	213,795	1.3747
10 Feb 2012 21:45:32	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	129,600	178,231	1.3752
09 Feb 2012 16:24:13	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	103,680	142,990	1.3791
06 Feb 2012 21:34:17	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	77,760	106,870	1.3744
03 Feb 2012 15:42:52	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	51,840	70,822	1.3662
01 Feb 2012 21:25:15	1193948	14032407	hadcm3n_u2p3_1980_40_007742906_2	25,920	35,287	1.3614