Task 12748569

Name	hadcm3n_o6fu_1900_40_007203677_0
Workunit	7401957
Created	28 Mar 2011, 14:17:10 UTC
Sent	29 Mar 2011, 4:01:47 UTC
Report deadline	28 Jun 2011, 11:28:58 UTC
Received	26 Apr 2011, 5:57:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1100163
Run time	8 days 12 hours 42 min 22 sec
CPU time	8 days 6 hours 31 min 26 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:25:16 (8576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:08:10 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:43:37 (6520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:55:05 (8272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:55:07 (8272): No heartbeat from core client for 30 sec - exiting 14:55:08 (8272): No heartbeat from core client for 30 sec - exiting 16:31:00 (8964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:32:17 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:50:14 (1632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:25:42 (7276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:44:27 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:56:43 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:56:44 (4904): No heartbeat from core client for 30 sec - exiting 14:56:45 (4904): No heartbeat from core client for 30 sec - exiting 14:56:46 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:56:41 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:04:17 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:54:41 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:23:08 (1952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:24 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:28 (5440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:26:48 (6180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:16:30 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:22:07 (1612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:44:48 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:17 (4152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:35 (4176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:53:12 (7312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:37:48 (896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:34 (8172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:42 (8172): No heartbeat from core client for 30 sec - exiting 04:50:33 (4024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:54:03 (1868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:36:45 (7708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:51:26 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:10:22 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:28 (9048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:41:43 (2684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:20:38 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:31 (7980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:30:42 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:26 (7512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:31 (9056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:17 (8884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:53:20 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:30 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:24 (8552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:34 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:33:19 (8008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:33:20 (8008): No heartbeat from core client for 30 sec - exiting 17:26:27 (8044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:25:25 (3704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:04:29 (2992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:19 (8744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:04:28 (8060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:35:45 (9888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:02 (9728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:55:07 (8656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:37:36 (9828): No heartbeat from core client for 30 sec - exiting 09:37:37 (9828): No heartbeat from core client for 30 sec - exiting 09:37:38 (9828): No heartbeat from core client for 30 sec - exiting 09:37:39 (9828): No heartbeat from core client for 30 sec - exiting 09:37:40 (9828): No heartbeat from core client for 30 sec - exiting 09:37:41 (9828): No heartbeat from core client for 30 sec - exiting 09:37:42 (9828): No heartbeat from core client for 30 sec - exiting 09:37:43 (9828): No heartbeat from core client for 30 sec - exiting 09:37:44 (9828): No heartbeat from core client for 30 sec - exiting 09:37:45 (9828): No heartbeat from core client for 30 sec - exiting 09:37:46 (9828): No heartbeat from core client for 30 sec - exiting 09:37:47 (9828): No heartbeat from core client for 30 sec - exiting 09:37:48 (9828): No heartbeat from core client for 30 sec - exiting 09:37:49 (9828): No heartbeat from core client for 30 sec - exiting 09:37:50 (9828): No heartbeat from core client for 30 sec - exiting 09:37:51 (9828): No heartbeat from core client for 30 sec - exiting 09:37:52 (9828): No heartbeat from core client for 30 sec - exiting 09:37:53 (9828): No heartbeat from core client for 30 sec - exiting 09:37:54 (9828): No heartbeat from core client for 30 sec - exiting 09:37:55 (9828): No heartbeat from core client for 30 sec - exiting 09:37:56 (9828): No heartbeat from core client for 30 sec - exiting 09:37:57 (9828): No heartbeat from core client for 30 sec - exiting 09:37:58 (9828): No heartbeat from core client for 30 sec - exiting 09:37:59 (9828): No heartbeat from core client for 30 sec - exiting 09:38:00 (9828): No heartbeat from core client for 30 sec - exiting 09:38:01 (9828): No heartbeat from core client for 30 sec - exiting 09:38:02 (9828): No heartbeat from core client for 30 sec - exiting 09:38:03 (9828): No heartbeat from core client for 30 sec - exiting 09:38:04 (9828): No heartbeat from core client for 30 sec - exiting 09:38:05 (9828): No heartbeat from core client for 30 sec - exiting 09:38:06 (9828): No heartbeat from core client for 30 sec - exiting 09:38:07 (9828): No heartbeat from core client for 30 sec - exiting 09:38:08 (9828): No heartbeat from core client for 30 sec - exiting 09:38:09 (9828): No heartbeat from core client for 30 sec - exiting 09:38:10 (9828): No heartbeat from core client for 30 sec - exiting 09:38:11 (9828): No heartbeat from core client for 30 sec - exiting 09:38:12 (9828): No heartbeat from core client for 30 sec - exiting 09:38:13 (9828): No heartbeat from core client for 30 sec - exiting 09:38:14 (9828): No heartbeat from core client for 30 sec - exiting 09:38:15 (9828): No heartbeat from core client for 30 sec - exiting 09:38:16 (9828): No heartbeat from core client for 30 sec - exiting 09:38:17 (9828): No heartbeat from core client for 30 sec - exiting 09:38:18 (9828): No heartbeat from core client for 30 sec - exiting 09:38:19 (9828): No heartbeat from core client for 30 sec - exiting 09:38:20 (9828): No heartbeat from core client for 30 sec - exiting 09:38:21 (9828): No heartbeat from core client for 30 sec - exiting 09:38:22 (9828): No heartbeat from core client for 30 sec - exiting 09:38:23 (9828): No heartbeat from core client for 30 sec - exiting 09:38:24 (9828): No heartbeat from core client for 30 sec - exiting 09:38:25 (9828): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5896, selfPID=5896, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o6fuko.pja8c10 Error converting file to netcdf: dataout/o6fuko.pia8c10 Error converting file to netcdf: dataout/o6fuko.pfa8c10 Error converting file to netcdf: dataout/o6fuka.pha8c10 Error converting file to netcdf: dataout/o6fuka.pga8c10 Error converting file to netcdf: dataout/o6fuka.pea8c10 12:53:00 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:57:06 (9956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:27:21 (9996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:12:51 (10160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:36 (9028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:37 (9028): No heartbeat from core client for 30 sec - exiting 14:23:38 (9028): No heartbeat from core client for 30 sec - exiting 14:23:39 (9028): No heartbeat from core client for 30 sec - exiting 14:23:40 (9028): No heartbeat from core client for 30 sec - exiting 14:23:41 (9028): No heartbeat from core client for 30 sec - exiting 14:23:43 (9028): No heartbeat from core client for 30 sec - exiting 14:23:44 (9028): No heartbeat from core client for 30 sec - exiting 14:23:45 (9028): No heartbeat from core client for 30 sec - exiting 14:23:46 (9028): No heartbeat from core client for 30 sec - exiting 17:34:31 (848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7712, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 10:48:07 (6428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:11:57 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:58 (5436): No heartbeat from core client for 30 sec - exiting 05:28:06 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:28 (3380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:09 (6756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:23 (6268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:23:07 (6384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:26:41 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:06 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:30 (6880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:51 (7148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:45:04 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:05:00 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:53:03 (3152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:50 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:55 (7048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:19:23 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:45:03 (7316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:50 (7280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:51 (7280): No heartbeat from core client for 30 sec - exiting 09:50:52 (7280): No heartbeat from core client for 30 sec - exiting 09:50:53 (7280): No heartbeat from core client for 30 sec - exiting 09:52:45 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:08 (7176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76CE3232 read attempt to address 0x40C714DA Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o6fu_1900_40_007203677/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Apr 2011 15:54:09	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	259,200	714,678	2.7572
21 Apr 2011 04:53:36	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	233,280	644,885	2.7644
11 Apr 2011 17:14:22	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	207,360	575,835	2.7770
10 Apr 2011 06:51:03	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	181,440	498,393	2.7469
09 Apr 2011 03:44:12	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	155,520	424,417	2.7290
08 Apr 2011 03:41:49	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	129,600	352,518	2.7200
06 Apr 2011 19:34:47	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	103,680	282,219	2.7220
05 Apr 2011 15:41:50	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	77,760	213,137	2.7410
04 Apr 2011 17:00:31	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	51,840	143,358	2.7654
31 Mar 2011 06:52:54	1100163	12748569	hadcm3n_o6fu_1900_40_007203677_0	25,920	71,620	2.7631