Task 14366732

Name	hadcm3n_103s_1980_40_007858501_1
Workunit	8013613
Created	5 Apr 2012, 19:24:10 UTC
Sent	5 Apr 2012, 19:43:53 UTC
Report deadline	6 Jul 2012, 3:11:04 UTC
Received	6 May 2012, 20:13:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1193660
Run time	4 days 18 hours 52 min 37 sec
CPU time	4 days 14 hours 52 min
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:03:41 (5384): No heartbeat from core client for 30 sec - exiting 02:03:42 (5384): No heartbeat from core client for 30 sec - exiting 02:03:43 (5384): No heartbeat from core client for 30 sec - exiting 02:03:44 (5384): No heartbeat from core client for 30 sec - exiting 02:03:45 (5384): No heartbeat from core client for 30 sec - exiting 02:03:46 (5384): No heartbeat from core client for 30 sec - exiting 02:03:47 (5384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:48 (5384): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:00:38 (4652): No heartbeat from core client for 30 sec - exiting 09:00:39 (4652): No heartbeat from core client for 30 sec - exiting 09:00:40 (4652): No heartbeat from core client for 30 sec - exiting 09:00:41 (4652): No heartbeat from core client for 30 sec - exiting 09:00:42 (4652): No heartbeat from core client for 30 sec - exiting 09:00:43 (4652): No heartbeat from core client for 30 sec - exiting 09:00:44 (4652): No heartbeat from core client for 30 sec - exiting 09:00:45 (4652): No heartbeat from core client for 30 sec - exiting 09:00:46 (4652): No heartbeat from core client for 30 sec - exiting 09:00:47 (4652): No heartbeat from core client for 30 sec - exiting 09:00:48 (4652): No heartbeat from core client for 30 sec - exiting 09:00:49 (4652): No heartbeat from core client for 30 sec - exiting 09:00:50 (4652): No heartbeat from core client for 30 sec - exiting 09:00:51 (4652): No heartbeat from core client for 30 sec - exiting 09:00:52 (4652): No heartbeat from core client for 30 sec - exiting 09:00:53 (4652): No heartbeat from core client for 30 sec - exiting 09:00:54 (4652): No heartbeat from core client for 30 sec - exiting 09:00:55 (4652): No heartbeat from core client for 30 sec - exiting 09:00:56 (4652): No heartbeat from core client for 30 sec - exiting 09:00:57 (4652): No heartbeat from core client for 30 sec - exiting 09:00:58 (4652): No heartbeat from core client for 30 sec - exiting 09:00:59 (4652): No heartbeat from core client for 30 sec - exiting 09:01:00 (4652): No heartbeat from core client for 30 sec - exiting 09:01:01 (4652): No heartbeat from core client for 30 sec - exiting 09:01:02 (4652): No heartbeat from core client for 30 sec - exiting 09:01:03 (4652): No heartbeat from core client for 30 sec - exiting 09:01:04 (4652): No heartbeat from core client for 30 sec - exiting 09:01:05 (4652): No heartbeat from core client for 30 sec - exiting 09:01:06 (4652): No heartbeat from core client for 30 sec - exiting 09:01:07 (4652): No heartbeat from core client for 30 sec - exiting 09:01:08 (4652): No heartbeat from core client for 30 sec - exiting 09:01:09 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:53:06 (4636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:31:09 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:51:55 (3104): No heartbeat from core client for 30 sec - exiting 08:51:56 (3104): No heartbeat from core client for 30 sec - exiting 08:51:57 (3104): No heartbeat from core client for 30 sec - exiting 08:51:58 (3104): No heartbeat from core client for 30 sec - exiting 08:51:59 (3104): No heartbeat from core client for 30 sec - exiting 08:52:00 (3104): No heartbeat from core client for 30 sec - exiting 08:52:01 (3104): No heartbeat from core client for 30 sec - exiting 08:52:02 (3104): No heartbeat from core client for 30 sec - exiting 08:52:03 (3104): No heartbeat from core client for 30 sec - exiting 08:52:04 (3104): No heartbeat from core client for 30 sec - exiting 08:52:05 (3104): No heartbeat from core client for 30 sec - exiting 08:52:06 (3104): No heartbeat from core client for 30 sec - exiting 08:52:07 (3104): No heartbeat from core client for 30 sec - exiting 08:52:08 (3104): No heartbeat from core client for 30 sec - exiting 08:52:09 (3104): No heartbeat from core client for 30 sec - exiting 08:52:10 (3104): No heartbeat from core client for 30 sec - exiting 08:52:11 (3104): No heartbeat from core client for 30 sec - exiting 08:52:12 (3104): No heartbeat from core client for 30 sec - exiting 08:52:13 (3104): No heartbeat from core client for 30 sec - exiting 08:52:14 (3104): No heartbeat from core client for 30 sec - exiting 08:52:15 (3104): No heartbeat from core client for 30 sec - exiting 08:52:16 (3104): No heartbeat from core client for 30 sec - exiting 08:52:17 (3104): No heartbeat from core client for 30 sec - exiting 08:52:18 (3104): No heartbeat from core client for 30 sec - exiting 08:52:20 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:21 (3104): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4968, selfPID=4968, iMonCtr=1 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_103s_1980_40_007858501/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 May 2012 12:38:34	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	259,200	399,192	1.5401
05 May 2012 20:01:59	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	233,280	360,376	1.5448
05 May 2012 08:48:11	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	207,360	321,620	1.5510
04 May 2012 17:04:14	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	181,440	282,629	1.5577
03 May 2012 14:01:15	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	155,520	243,039	1.5628
01 May 2012 08:06:16	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	129,600	203,717	1.5719
30 Apr 2012 10:11:01	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	103,680	164,678	1.5883
26 Apr 2012 19:28:34	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	77,760	124,526	1.6014
25 Apr 2012 12:52:57	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	51,840	83,910	1.6186
23 Apr 2012 15:17:01	1193660	14366732	hadcm3n_103s_1980_40_007858501_1	25,920	41,999	1.6203