Task 15874204

Name	hadcm3n_u3d9_2020_40_008339067_3
Workunit	8489928
Created	30 Jun 2013, 14:26:30 UTC
Sent	30 Jun 2013, 15:24:58 UTC
Report deadline	29 Sep 2013, 22:52:09 UTC
Received	23 Jul 2013, 0:03:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	722346
Run time	4 days 4 hours 23 min 34 sec
CPU time	4 days 0 hours 36 min 15 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.58 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 16:33:19 (7048): No heartbeat from core client for 30 sec - exiting 16:33:20 (7048): No heartbeat from core client for 30 sec - exiting 16:33:21 (7048): No heartbeat from core client for 30 sec - exiting 16:33:22 (7048): No heartbeat from core client for 30 sec - exiting 16:33:23 (7048): No heartbeat from core client for 30 sec - exiting 16:33:25 (7048): No heartbeat from core client for 30 sec - exiting 16:33:26 (7048): No heartbeat from core client for 30 sec - exiting 16:33:27 (7048): No heartbeat from core client for 30 sec - exiting 16:33:28 (7048): No heartbeat from core client for 30 sec - exiting 16:33:29 (7048): No heartbeat from core client for 30 sec - exiting 16:33:30 (7048): No heartbeat from core client for 30 sec - exiting 16:33:31 (7048): No heartbeat from core client for 30 sec - exiting 16:33:32 (7048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:10:15 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:13:54 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:44:44 (2376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:04:27 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:01:18 (8008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:46:15 (7284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:24:05 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/u3d9ko.pjm3c10 Error converting file to netcdf: dataout/u3d9ko.pim3c10 Error converting file to netcdf: dataout/u3d9ko.pfm3c10 Error converting file to netcdf: dataout/u3d9ka.phm3c10 Error converting file to netcdf: dataout/u3d9ka.pgm3c10 Error converting file to netcdf: dataout/u3d9ka.pem3c10 Error converting file to netcdf: dataout/u3d9ka.pdm3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:48:18 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:10:55 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:05:28 (3988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:28 (6180): No heartbeat from core client for 30 sec - exiting 09:20:29 (6180): No heartbeat from core client for 30 sec - exiting 09:20:30 (6180): No heartbeat from core client for 30 sec - exiting 09:20:31 (6180): No heartbeat from core client for 30 sec - exiting 09:20:32 (6180): No heartbeat from core client for 30 sec - exiting 09:20:33 (6180): No heartbeat from core client for 30 sec - exiting 09:20:34 (6180): No heartbeat from core client for 30 sec - exiting 09:20:36 (6180): No heartbeat from core client for 30 sec - exiting 09:20:37 (6180): No heartbeat from core client for 30 sec - exiting 09:20:38 (6180): No heartbeat from core client for 30 sec - exiting 09:20:39 (6180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:50:00 (6348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:11:04 (5452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... zip error: Could not create output file (was replacing the original zip file) cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file E:\BOINC/projects/climateprediction.net/hadcm3n_u3d9_2020_40_008339067/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 21:20:44	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	259,200	347,815	1.3419
23 Jul 2013 20:31:25	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	233,280	313,775	1.3451
23 Jul 2013 19:17:48	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	207,360	279,766	1.3492
23 Jul 2013 19:17:48	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	181,440	245,802	1.3547
23 Jul 2013 19:17:48	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	155,520	210,872	1.3559
23 Jul 2013 19:17:47	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	129,600	175,705	1.3557
10 Jul 2013 21:26:50	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	103,680	140,491	1.3550
07 Jul 2013 19:27:41	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	77,760	105,051	1.3510
04 Jul 2013 14:27:27	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	51,840	69,959	1.3495
02 Jul 2013 11:59:56	722346	15874204	hadcm3n_u3d9_2020_40_008339067_3	25,920	35,527	1.3706