Task 15798062

Name	hadcm3n_z90h_1960_40_008320998_1
Workunit	8472133
Created	27 May 2013, 0:17:54 UTC
Sent	27 May 2013, 0:18:02 UTC
Report deadline	26 Aug 2013, 7:45:13 UTC
Received	10 Jan 2014, 22:48:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1281494
Run time	19 days 8 hours 12 min 3 sec
CPU time	18 days 17 hours 47 min 19 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4316, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:10:35 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:36 (2100): No heartbeat from core client for 30 sec - exiting 21:10:37 (2100): No heartbeat from core client for 30 sec - exiting 21:10:38 (2100): No heartbeat from core client for 30 sec - exiting 21:10:39 (2100): No heartbeat from core client for 30 sec - exiting 21:10:40 (2100): No heartbeat from core client for 30 sec - exiting 21:10:41 (2100): No heartbeat from core client for 30 sec - exiting 21:10:42 (2100): No heartbeat from core client for 30 sec - exiting 21:10:43 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6900, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3172, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 11:35:37 (4192): No heartbeat from core client for 30 sec - exiting 11:35:39 (4192): No heartbeat from core client for 30 sec - exiting 11:35:40 (4192): No heartbeat from core client for 30 sec - exiting 11:35:41 (4192): No heartbeat from core client for 30 sec - exiting 11:35:42 (4192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1436, iMonCtr=1 Model crash detected, will try to restart... 15:45:21 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4380, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 17:23:37 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6024, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5936, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5288, iMonCtr=1 Model crash detected, will try to restart... 17:24:16 (5760): No heartbeat from core client for 30 sec - exiting 17:24:17 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6460, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4592, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6280, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 13:49:42 (6076): No heartbeat from core client for 30 sec - exiting 13:49:43 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3132, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3424, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5844, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 15:20:30 (5308): No heartbeat from core client for 30 sec - exiting 15:20:31 (5308): No heartbeat from core client for 30 sec - exiting 15:20:32 (5308): No heartbeat from core client for 30 sec - exiting 15:20:33 (5308): No heartbeat from core client for 30 sec - exiting 15:20:34 (5308): No heartbeat from core client for 30 sec - exiting 15:20:35 (5308): No heartbeat from core client for 30 sec - exiting 15:20:36 (5308): No heartbeat from core client for 30 sec - exiting 15:20:37 (5308): No heartbeat from core client for 30 sec - exiting 15:20:38 (5308): No heartbeat from core client for 30 sec - exiting 15:20:40 (5308): No heartbeat from core client for 30 sec - exiting 15:20:41 (5308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/z90hko.pji6c10 Error converting file to netcdf: dataout/z90hko.pii6c10 Error converting file to netcdf: dataout/z90hko.pfi6c10 Error converting file to netcdf: dataout/z90hka.phi6c10 Error converting file to netcdf: dataout/z90hka.pgi6c10 Error converting file to netcdf: dataout/z90hka.pei6c10 Error converting file to netcdf: dataout/z90hka.pdi6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1952, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:44:49 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3288, iMonCtr=1 Model crash detected, will try to restart... 13:05:16 (1504): No heartbeat from core client for 30 sec - exiting 13:05:18 (1504): No heartbeat from core client for 30 sec - exiting 13:05:19 (1504): No heartbeat from core client for 30 sec - exiting 13:05:20 (1504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:41 (5084): No heartbeat from core client for 30 sec - exiting 17:21:42 (5084): No heartbeat from core client for 30 sec - exiting 17:21:44 (5084): No heartbeat from core client for 30 sec - exiting 17:21:45 (5084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:28:42 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:39:23 (1496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:04:40 (5744): No heartbeat from core client for 30 sec - exiting 15:04:41 (5744): No heartbeat from core client for 30 sec - exiting 15:04:42 (5744): No heartbeat from core client for 30 sec - exiting 15:04:43 (5744): No heartbeat from core client for 30 sec - exiting 15:04:44 (5744): No heartbeat from core client for 30 sec - exiting 15:04:45 (5744): No heartbeat from core client for 30 sec - exiting 15:04:46 (5744): No heartbeat from core client for 30 sec - exiting 15:04:48 (5744): No heartbeat from core client for 30 sec - exiting 15:04:49 (5744): No heartbeat from core client for 30 sec - exiting 15:04:50 (5744): No heartbeat from core client for 30 sec - exiting 15:04:51 (5744): No heartbeat from core client for 30 sec - exiting 15:04:52 (5744): No heartbeat from core client for 30 sec - exiting 15:04:53 (5744): No heartbeat from core client for 30 sec - exiting 15:04:54 (5744): No heartbeat from core client for 30 sec - exiting 15:04:55 (5744): No heartbeat from core client for 30 sec - exiting 15:04:56 (5744): No heartbeat from core client for 30 sec - exiting 15:04:57 (5744): No heartbeat from core client for 30 sec - exiting 15:04:58 (5744): No heartbeat from core client for 30 sec - exiting 15:05:00 (5744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/z90hko.pjj4c10 Error converting file to netcdf: dataout/z90hko.pij4c10 Error converting file to netcdf: dataout/z90hko.pfj4c10 Error converting file to netcdf: dataout/z90hka.phj4c10 Error converting file to netcdf: dataout/z90hka.pgj4c10 Error converting file to netcdf: dataout/z90hka.pej4c10 Error converting file to netcdf: dataout/z90hka.pdj4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6940, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/z90hko.pjj8c10 Error converting file to netcdf: dataout/z90hko.pij8c10 Error converting file to netcdf: dataout/z90hko.pfj8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77363AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771F7383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771F7383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77957383 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_z90h_1960_40_008320998/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jan 2014 02:26:22	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	1,036,800	1,618,853	1.5614
21 Dec 2013 04:25:20	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	1,010,880	1,593,656	1.5765
16 Dec 2013 22:43:04	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	984,960	1,568,215	1.5922
15 Dec 2013 02:34:50	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	959,040	1,539,082	1.6048
09 Dec 2013 02:32:45	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	933,120	1,510,281	1.6185
07 Dec 2013 02:13:29	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	907,200	1,482,140	1.6338
26 Nov 2013 00:48:15	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	881,280	1,454,168	1.6501
05 Nov 2013 02:26:03	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	855,360	1,425,534	1.6666
03 Nov 2013 19:34:27	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	829,440	1,396,849	1.6841
02 Nov 2013 19:02:45	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	803,520	1,368,127	1.7027
27 Oct 2013 02:46:03	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	777,600	1,333,826	1.7153
22 Oct 2013 01:25:03	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	751,680	1,292,502	1.7195
12 Oct 2013 02:47:27	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	725,760	1,250,384	1.7229
03 Oct 2013 01:42:57	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	699,840	1,204,731	1.7214
27 Sep 2013 21:40:03	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	673,920	1,155,418	1.7145
21 Sep 2013 20:46:34	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	648,000	1,106,502	1.7076
16 Sep 2013 22:00:21	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	622,080	1,057,599	1.7001
14 Sep 2013 04:51:15	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	596,160	1,008,693	1.6920
10 Sep 2013 01:31:16	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	570,240	960,472	1.6843
02 Sep 2013 05:02:34	1281494	15798062	hadcm3n_z90h_1960_40_008320998_1	544,320	912,527	1.6765