Task 13610167

Name	hadcm3n_t12i_1940_40_007539834_1
Workunit	7737066
Created	6 Nov 2011, 4:08:08 UTC
Sent	9 Nov 2011, 3:23:10 UTC
Report deadline	8 Feb 2012, 10:50:21 UTC
Received	23 Dec 2011, 20:11:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179103
Run time	17 days 23 hours 44 min 43 sec
CPU time	13 days 8 hours 43 min 23 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 15:07:36 (2948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:47:53 (592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:00:09 (3912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:59:08 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:11:36 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2360, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 15:46:17 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:22 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:49:07 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:47:57 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:53 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:45:52 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:32 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:34 (4248): No heartbeat from core client for 30 sec - exiting 16:43:30 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:11 (376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:12 (376): No heartbeat from core client for 30 sec - exiting 23:41:07 (4192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:41:08 (4192): No heartbeat from core client for 30 sec - exiting 23:41:09 (4192): No heartbeat from core client for 30 sec - exiting 23:41:10 (4192): No heartbeat from core client for 30 sec - exiting 23:41:11 (4192): No heartbeat from core client for 30 sec - exiting 02:40:00 (4440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:51 (3492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:53 (3492): No heartbeat from core client for 30 sec - exiting 08:37:43 (4340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:33 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1352, iMonCtr=1 Model crash detected, will try to restart... 17:17:22 (3360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:29:53 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:28:45 (7960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:38:33 (2552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:26 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:36:21 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:14 (720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:10 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:04 (5420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:03 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:39:30 (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:30 (1932): No heartbeat from core client for 30 sec - exiting 18:38:31 (1932): No heartbeat from core client for 30 sec - exiting 18:38:33 (1932): No heartbeat from core client for 30 sec - exiting 18:38:34 (1932): No heartbeat from core client for 30 sec - exiting 18:38:35 (1932): No heartbeat from core client for 30 sec - exiting 18:38:36 (1932): No heartbeat from core client for 30 sec - exiting 18:38:37 (1932): No heartbeat from core client for 30 sec - exiting 18:38:38 (1932): No heartbeat from core client for 30 sec - exiting 18:38:39 (1932): No heartbeat from core client for 30 sec - exiting 18:38:40 (1932): No heartbeat from core client for 30 sec - exiting 18:38:42 (1932): No heartbeat from core client for 30 sec - exiting 18:38:43 (1932): No heartbeat from core client for 30 sec - exiting 18:38:44 (1932): No heartbeat from core client for 30 sec - exiting 18:38:45 (1932): No heartbeat from core client for 30 sec - exiting 18:38:46 (1932): No heartbeat from core client for 30 sec - exiting 18:38:47 (1932): No heartbeat from core client for 30 sec - exiting 18:38:48 (1932): No heartbeat from core client for 30 sec - exiting 18:38:49 (1932): No heartbeat from core client for 30 sec - exiting 18:38:50 (1932): No heartbeat from core client for 30 sec - exiting 18:38:51 (1932): No heartbeat from core client for 30 sec - exiting 18:38:53 (1932): No heartbeat from core client for 30 sec - exiting 18:38:54 (1932): No heartbeat from core client for 30 sec - exiting 18:38:55 (1932): No heartbeat from core client for 30 sec - exiting 18:38:56 (1932): No heartbeat from core client for 30 sec - exiting 18:38:57 (1932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t12iko.pje7c10 Error converting file to netcdf: dataout/t12iko.pie7c10 Error converting file to netcdf: dataout/t12iko.pfe7c10 Error converting file to netcdf: dataout/t12ika.phe7c10 Error converting file to netcdf: dataout/t12ika.pge7c10 Error converting file to netcdf: dataout/t12ika.pee7c10 Error converting file to netcdf: dataout/t12ika.pde7c10 21:36:55 (7764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:52 (2080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:07 (8176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:57 (6304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:58 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:48 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:51 (7280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:29:39 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:23 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:15:10 (944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:30:08 (16112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:29:08 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:49 (3328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:06:20 (2088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:11 (9528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:10 (8860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:02:54 (1232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:44 (4888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:32 (11412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:59:18 (9068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77946E0F read attempt to address 0x40794A9E Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_t12i_1940_40_007539834/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Dec 2011 20:12:14	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	259,200	1,154,592	4.4544
18 Dec 2011 19:38:18	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	233,280	1,039,672	4.4568
11 Dec 2011 21:45:10	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	207,360	922,356	4.4481
10 Dec 2011 00:49:08	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	181,440	804,665	4.4349
06 Dec 2011 16:42:27	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	155,520	694,960	4.4686
04 Dec 2011 20:42:25	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	129,600	585,562	4.5182
01 Dec 2011 18:52:00	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	103,680	467,387	4.5080
26 Nov 2011 16:26:25	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	77,760	348,832	4.4860
22 Nov 2011 23:15:10	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	51,840	229,884	4.4345
15 Nov 2011 17:37:28	1179103	13610167	hadcm3n_t12i_1940_40_007539834_1	25,920	113,146	4.3652