Task 12984708

Name	hadcm3n_p53f_1900_40_007224187_2
Workunit	7422427
Created	17 Jun 2011, 0:12:48 UTC
Sent	17 Jun 2011, 0:13:05 UTC
Report deadline	16 Sep 2011, 7:40:16 UTC
Received	1 Jul 2011, 6:21:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1100209
Run time	12 days 22 hours 23 min
CPU time	12 days 16 hours 30 min 10 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6052, iMonCtr=1 Model crash detected, will try to restart... 03:32:35 (3152): No heartbeat from core client for 30 sec - exiting 03:32:36 (3152): No heartbeat from core client for 30 sec - exiting 03:32:37 (3152): No heartbeat from core client for 30 sec - exiting 03:32:38 (3152): No heartbeat from core client for 30 sec - exiting 03:32:39 (3152): No heartbeat from core client for 30 sec - exiting 03:32:40 (3152): No heartbeat from core client for 30 sec - exiting 03:32:41 (3152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:41:19 (4004): No heartbeat from core client for 30 sec - exiting 03:41:20 (4004): No heartbeat from core client for 30 sec - exiting 03:41:21 (4004): No heartbeat from core client for 30 sec - exiting 03:41:22 (4004): No heartbeat from core client for 30 sec - exiting 03:41:23 (4004): No heartbeat from core client for 30 sec - exiting 03:41:24 (4004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:25 (4004): No heartbeat from core client for 30 sec - exiting 16:25:31 (2488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:26:10 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:30 (156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:26 (3612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:46 (2860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:39 (980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:25:21 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:28:51 (2488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:28:52 (2488): No heartbeat from core client for 30 sec - exiting 12:15:09 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:20:02 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:32 (3236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:33 (3236): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/p53fko.pja3c10 Error converting file to netcdf: dataout/p53fko.pia3c10 Error converting file to netcdf: dataout/p53fko.pfa3c10 Error converting file to netcdf: dataout/p53fka.pha3c10 Error converting file to netcdf: dataout/p53fka.pga3c10 Error converting file to netcdf: dataout/p53fka.pea3c10 Error converting file to netcdf: dataout/p53fka.pda3c10 22:10:31 (2760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:32 (2760): No heartbeat from core client for 30 sec - exiting 00:14:28 (2464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:29 (2464): No heartbeat from core client for 30 sec - exiting 03:20:03 (3460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:22 (2316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:59:46 (2084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:59:48 (2084): No heartbeat from core client for 30 sec - exiting 07:54:38 (2996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:29:04 (1436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:36:22 (3024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:33:03 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:15 (1300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:17 (1300): No heartbeat from core client for 30 sec - exiting 13:40:18 (1300): No heartbeat from core client for 30 sec - exiting 13:40:19 (1300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:27:25 (2132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:32:07 (2316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:25:12 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:46:00 (2884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:00 (1156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:12:03 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:11 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:12 (2936): No heartbeat from core client for 30 sec - exiting 04:02:19 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:13:30 (2700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:16:09 (1224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:19:30 (3632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:59 (4008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:22 (796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:23 (796): No heartbeat from core client for 30 sec - exiting 21:32:43 (5892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:30 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:34:31 (5328): No heartbeat from core client for 30 sec - exiting 02:39:43 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:44 (4224): No heartbeat from core client for 30 sec - exiting 05:15:06 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:21 (3564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:06 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:07 (5060): No heartbeat from core client for 30 sec - exiting 15:06:16 (5908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:11:05 (5392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:04 (5248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:05 (5248): No heartbeat from core client for 30 sec - exiting 16:19:11 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:19:13 (4260): No heartbeat from core client for 30 sec - exiting 21:23:31 (6020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:30:59 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:00 (2784): No heartbeat from core client for 30 sec - exiting 03:31:01 (2784): No heartbeat from core client for 30 sec - exiting 03:31:03 (2784): No heartbeat from core client for 30 sec - exiting 03:31:04 (2784): No heartbeat from core client for 30 sec - exiting 03:31:05 (2784): No heartbeat from core client for 30 sec - exiting 03:31:06 (2784): No heartbeat from core client for 30 sec - exiting 03:31:07 (2784): No heartbeat from core client for 30 sec - exiting 03:31:08 (2784): No heartbeat from core client for 30 sec - exiting 03:31:09 (2784): No heartbeat from core client for 30 sec - exiting 03:31:10 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 03:40:20 (2788): No heartbeat from core client for 30 sec - exiting 03:40:21 (2788): No heartbeat from core client for 30 sec - exiting 03:40:22 (2788): No heartbeat from core client for 30 sec - exiting 03:40:23 (2788): No heartbeat from core client for 30 sec - exiting 03:40:24 (2788): No heartbeat from core client for 30 sec - exiting 03:40:25 (2788): No heartbeat from core client for 30 sec - exiting 03:40:26 (2788): No heartbeat from core client for 30 sec - exiting 03:40:27 (2788): No heartbeat from core client for 30 sec - exiting 03:40:28 (2788): No heartbeat from core client for 30 sec - exiting 03:40:30 (2788): No heartbeat from core client for 30 sec - exiting 03:40:31 (2788): No heartbeat from core client for 30 sec - exiting 03:40:32 (2788): No heartbeat from core client for 30 sec - exiting 03:40:33 (2788): No heartbeat from core client for 30 sec - exiting 03:40:34 (2788): No heartbeat from core client for 30 sec - exiting 03:40:35 (2788): No heartbeat from core client for 30 sec - exiting 03:40:36 (2788): No heartbeat from core client for 30 sec - exiting 03:40:37 (2788): No heartbeat from core client for 30 sec - exiting 03:40:38 (2788): No heartbeat from core client for 30 sec - exiting 03:40:39 (2788): No heartbeat from core client for 30 sec - exiting 03:40:40 (2788): No heartbeat from core client for 30 sec - exiting 03:40:41 (2788): No heartbeat from core client for 30 sec - exiting 03:40:42 (2788): No heartbeat from core client for 30 sec - exiting 03:40:43 (2788): No heartbeat from core client for 30 sec - exiting 03:40:44 (2788): No heartbeat from core client for 30 sec - exiting 03:40:45 (2788): No heartbeat from core client for 30 sec - exiting 03:40:46 (2788): No heartbeat from core client for 30 sec - exiting 03:40:47 (2788): No heartbeat from core client for 30 sec - exiting 03:40:48 (2788): No heartbeat from core client for 30 sec - exiting 03:40:49 (2788): No heartbeat from core client for 30 sec - exiting 03:40:50 (2788): No heartbeat from core client for 30 sec - exiting 03:40:51 (2788): No heartbeat from core client for 30 sec - exiting 03:40:52 (2788): No heartbeat from core client for 30 sec - exiting 03:40:53 (2788): No heartbeat from core client for 30 sec - exiting 03:40:54 (2788): No heartbeat from core client for 30 sec - exiting 03:40:55 (2788): No heartbeat from core client for 30 sec - exiting 03:40:56 (2788): No heartbeat from core client for 30 sec - exiting 03:40:57 (2788): No heartbeat from core client for 30 sec - exiting 03:40:58 (2788): No heartbeat from core client for 30 sec - exiting 03:40:59 (2788): No heartbeat from core client for 30 sec - exiting 03:41:00 (2788): No heartbeat from core client for 30 sec - exiting 03:41:01 (2788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:02 (2788): No heartbeat from core client for 30 sec - exiting 05:07:39 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:34 (2876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:09 (676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:11:23 (500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:00 (3204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3600, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:17:46 (2252): No heartbeat from core client for 30 sec - exiting 22:17:47 (2252): No heartbeat from core client for 30 sec - exiting 22:17:48 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x774D6E0F read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_p53f_1900_40_007224187/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Jul 2011 06:25:12	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	518,400	1,096,193	2.1146
30 Jun 2011 12:06:58	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	492,480	1,039,653	2.1111
29 Jun 2011 19:54:18	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	466,560	983,357	2.1077
29 Jun 2011 02:05:39	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	440,640	929,290	2.1090
28 Jun 2011 09:59:28	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	414,720	873,132	2.1054
27 Jun 2011 16:11:18	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	388,800	817,680	2.1031
26 Jun 2011 22:30:52	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	362,880	764,461	2.1066
26 Jun 2011 05:11:27	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	336,960	710,971	2.1100
25 Jun 2011 03:55:35	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	311,040	655,825	2.1085
24 Jun 2011 12:08:17	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	285,120	599,479	2.1025
23 Jun 2011 18:15:25	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	259,200	543,573	2.0971
23 Jun 2011 00:23:58	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	233,280	487,262	2.0887
22 Jun 2011 08:06:58	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	207,360	431,434	2.0806
21 Jun 2011 16:14:34	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	181,440	375,355	2.0688
21 Jun 2011 00:39:41	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	155,520	321,283	2.0659
20 Jun 2011 10:10:44	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	129,600	269,754	2.0814
19 Jun 2011 23:10:23	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	103,680	213,847	2.0626
19 Jun 2011 23:10:23	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	77,760	160,135	2.0593
19 Jun 2011 22:02:42	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	51,840	107,235	2.0686
17 Jun 2011 15:38:11	1100209	12984708	hadcm3n_p53f_1900_40_007224187_2	25,920	52,701	2.0332