Task 15700024

Name	hadcm3n_u3g3_2020_40_008338786_1
Workunit	8489647
Created	2 Apr 2013, 18:07:36 UTC
Sent	2 Apr 2013, 18:07:52 UTC
Report deadline	3 Jul 2013, 1:35:03 UTC
Received	5 May 2013, 19:48:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1231665
Run time	14 days 5 hours 28 min 47 sec
CPU time	14 days 5 hours 28 min 47 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	1.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>5.2.13</core_client_version> <message>The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 08:44:58 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:05 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:15 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:25 (5008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:36 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:46 (4584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:51:57 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:53:05 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:12 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:24 (408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:56:34 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:44 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:54 (668): No heartbeat from core client for 30 sec - exiting 08:58:56 (668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:57 (668): No heartbeat from core client for 30 sec - exiting 09:00:04 (1820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:14 (340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:02:25 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 07:13:20 (4268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:07:06 (3792): No heartbeat from core client for 30 sec - exiting 07:07:07 (3792): No heartbeat from core client for 30 sec - exiting 07:07:09 (3792): No heartbeat from core client for 30 sec - exiting 07:07:10 (3792): No heartbeat from core client for 30 sec - exiting 07:07:11 (3792): No heartbeat from core client for 30 sec - exiting 07:07:12 (3792): No heartbeat from core client for 30 sec - exiting 07:07:13 (3792): No heartbeat from core client for 30 sec - exiting 07:07:14 (3792): No heartbeat from core client for 30 sec - exiting 07:07:16 (3792): No heartbeat from core client for 30 sec - exiting 07:07:17 (3792): No heartbeat from core client for 30 sec - exiting 07:07:18 (3792): No heartbeat from core client for 30 sec - exiting 07:07:19 (3792): No heartbeat from core client for 30 sec - exiting 07:07:20 (3792): No heartbeat from core client for 30 sec - exiting 07:07:21 (3792): No heartbeat from core client for 30 sec - exiting 07:07:22 (3792): No heartbeat from core client for 30 sec - exiting 07:07:23 (3792): No heartbeat from core client for 30 sec - exiting 07:07:24 (3792): No heartbeat from core client for 30 sec - exiting 07:07:25 (3792): No heartbeat from core client for 30 sec - exiting 07:07:27 (3792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u3g3ko.pjm6c10 Error converting file to netcdf: dataout/u3g3ko.pim6c10 Error converting file to netcdf: dataout/u3g3ko.pfm6c10 Error converting file to netcdf: dataout/u3g3ka.phm6c10 Error converting file to netcdf: dataout/u3g3ka.pgm6c10 Error converting file to netcdf: dataout/u3g3ka.pem6c10 Error converting file to netcdf: dataout/u3g3ka.pdm6c10 06:59:29 (5388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:30 (5388): No heartbeat from core client for 30 sec - exiting Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... OPEN: Unable to Open File dataout/u3g3ko.dam8ab0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 Model crashed: jobs/climate.cpdc diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 12:46:20 (4168): Can't open init data file - running in standalone mode 12:46:20 (4168): start_timer_thread(): CreateThread() failed, errno 2 Model crashed: dataout/stdout_um.txt diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 12:46:27 (6084): Can't open init data file - running in standalone mode 12:46:27 (6084): start_timer_thread(): CreateThread() failed, errno 2 Model crashed: dataout/stdout_um.txt Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2548, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2548, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Apr 2013 18:37:17	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	181,440	1,109,847	6.1169
26 Apr 2013 14:15:20	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	155,520	951,565	6.1186
22 Apr 2013 01:06:12	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	129,600	788,018	6.0804
17 Apr 2013 23:46:40	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	103,680	625,428	6.0323
13 Apr 2013 20:06:07	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	77,760	461,580	5.9360
09 Apr 2013 20:07:02	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	51,840	300,815	5.8028
05 Apr 2013 18:58:44	1231665	15700024	hadcm3n_u3g3_2020_40_008338786_1	25,920	143,301	5.5286