Task 13129204

Name	hadcm3n_ymzz_1900_40_007362665_1
Workunit	7560095
Created	6 Jul 2011, 15:27:08 UTC
Sent	6 Jul 2011, 18:12:33 UTC
Report deadline	6 Oct 2011, 1:39:44 UTC
Received	18 Jul 2011, 18:39:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1142958
Run time	2 days 13 hours 32 min 45 sec
CPU time	2 days 3 hours 47 min 9 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.43 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6468, selfPID=6468, iMonCtr=1 10:13:37 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:13:47 (4128): No heartbeat from core client for 30 sec - exiting 11:13:48 (4128): No heartbeat from core client for 30 sec - exiting 11:13:49 (4128): No heartbeat from core client for 30 sec - exiting 11:13:50 (4128): No heartbeat from core client for 30 sec - exiting 11:13:51 (4128): No heartbeat from core client for 30 sec - exiting 11:13:52 (4128): No heartbeat from core client for 30 sec - exiting 11:13:53 (4128): No heartbeat from core client for 30 sec - exiting 11:13:54 (4128): No heartbeat from core client for 30 sec - exiting 11:13:55 (4128): No heartbeat from core client for 30 sec - exiting 11:13:56 (4128): No heartbeat from core client for 30 sec - exiting 11:13:57 (4128): No heartbeat from core client for 30 sec - exiting 11:13:58 (4128): No heartbeat from core client for 30 sec - exiting 11:13:59 (4128): No heartbeat from core client for 30 sec - exiting 11:14:00 (4128): No heartbeat from core client for 30 sec - exiting 11:14:01 (4128): No heartbeat from core client for 30 sec - exiting 11:14:02 (4128): No heartbeat from core client for 30 sec - exiting 11:14:03 (4128): No heartbeat from core client for 30 sec - exiting 11:14:04 (4128): No heartbeat from core client for 30 sec - exiting 11:14:05 (4128): No heartbeat from core client for 30 sec - exiting 11:14:06 (4128): No heartbeat from core client for 30 sec - exiting 11:14:07 (4128): No heartbeat from core client for 30 sec - exiting 11:14:08 (4128): No heartbeat from core client for 30 sec - exiting 11:14:09 (4128): No heartbeat from core client for 30 sec - exiting 11:14:10 (4128): No heartbeat from core client for 30 sec - exiting 11:14:11 (4128): No heartbeat from core client for 30 sec - exiting 11:14:12 (4128): No heartbeat from core client for 30 sec - exiting 11:14:13 (4128): No heartbeat from core client for 30 sec - exiting 11:14:14 (4128): No heartbeat from core client for 30 sec - exiting 11:14:15 (4128): No heartbeat from core client for 30 sec - exiting 11:14:16 (4128): No heartbeat from core client for 30 sec - exiting 11:14:17 (4128): No heartbeat from core client for 30 sec - exiting 11:14:18 (4128): No heartbeat from core client for 30 sec - exiting 11:14:19 (4128): No heartbeat from core client for 30 sec - exiting 11:14:20 (4128): No heartbeat from core client for 30 sec - exiting 11:14:21 (4128): No heartbeat from core client for 30 sec - exiting 11:14:22 (4128): No heartbeat from core client for 30 sec - exiting 11:14:24 (4128): No heartbeat from core client for 30 sec - exiting 11:14:25 (4128): No heartbeat from core client for 30 sec - exiting 11:14:26 (4128): No heartbeat from core client for 30 sec - exiting 11:14:27 (4128): No heartbeat from core client for 30 sec - exiting 11:14:28 (4128): No heartbeat from core client for 30 sec - exiting 11:14:29 (4128): No heartbeat from core client for 30 sec - exiting 11:14:30 (4128): No heartbeat from core client for 30 sec - exiting 11:14:31 (4128): No heartbeat from core client for 30 sec - exiting 11:14:32 (4128): No heartbeat from core client for 30 sec - exiting 11:14:33 (4128): No heartbeat from core client for 30 sec - exiting 11:14:34 (4128): No heartbeat from core client for 30 sec - exiting 11:14:35 (4128): No heartbeat from core client for 30 sec - exiting 11:14:36 (4128): No heartbeat from core client for 30 sec - exiting 11:14:37 (4128): No heartbeat from core client for 30 sec - exiting 11:14:38 (4128): No heartbeat from core client for 30 sec - exiting 11:14:39 (4128): No heartbeat from core client for 30 sec - exiting 11:14:40 (4128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:03:05 (3872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:18 (3872): No heartbeat from core client for 30 sec - exiting 15:03:19 (3872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:39 (6084): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... 13:40:39 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:40 (6784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6892, selfPID=6892, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ymzzko.pja3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:30:23 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:24 (4780): No heartbeat from core client for 30 sec - exiting 09:30:25 (4780): No heartbeat from core client for 30 sec - exiting 09:30:26 (4780): No heartbeat from core client for 30 sec - exiting 09:30:27 (4780): No heartbeat from core client for 30 sec - exiting 09:30:28 (4780): No heartbeat from core client for 30 sec - exiting 09:30:29 (4780): No heartbeat from core client for 30 sec - exiting 09:30:30 (4780): No heartbeat from core client for 30 sec - exiting 09:30:31 (4780): No heartbeat from core client for 30 sec - exiting 09:30:32 (4780): No heartbeat from core client for 30 sec - exiting 09:30:33 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5276, selfPID=5276, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ymzzko.pja4c10 Error converting file to netcdf: dataout/ymzzko.pia4c10 Error converting file to netcdf: dataout/ymzzko.pfa4c10 Error converting file to netcdf: dataout/ymzzka.pha4c10 Error converting file to netcdf: dataout/ymzzka.pga4c10 Error converting file to netcdf: dataout/ymzzka.pea4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 14:32:02	1142958	13129204	hadcm3n_ymzz_1900_40_007362665_1	103,680	168,268	1.6230
25 Jul 2011 14:22:08	1142958	13129204	hadcm3n_ymzz_1900_40_007362665_1	77,760	125,260	1.6109
25 Jul 2011 14:22:08	1142958	13129204	hadcm3n_ymzz_1900_40_007362665_1	51,840	83,974	1.6199
08 Jul 2011 18:22:49	1142958	13129204	hadcm3n_ymzz_1900_40_007362665_1	25,920	42,379	1.6350