Task 15997441

Name	hadcm3n_n35w_1880_40_008374073_1
Workunit	8524932
Created	31 Aug 2013, 1:15:36 UTC
Sent	31 Aug 2013, 1:21:29 UTC
Report deadline	30 Nov 2013, 8:48:40 UTC
Received	19 Oct 2013, 3:45:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1288622
Run time	13 days 0 hours 12 min 57 sec
CPU time	12 days 11 hours 5 min 14 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> デバイスがコマンドを認識できません。 (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 07:29:47 (1780): No heartbeat from core client for 30 sec - exiting 07:29:48 (1780): No heartbeat from core client for 30 sec - exiting 07:29:49 (1780): No heartbeat from core client for 30 sec - exiting 07:29:50 (1780): No heartbeat from core client for 30 sec - exiting 07:29:51 (1780): No heartbeat from core client for 30 sec - exiting 07:29:52 (1780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:10:42 (3340): No heartbeat from core client for 30 sec - exiting 19:10:43 (3340): No heartbeat from core client for 30 sec - exiting 19:10:44 (3340): No heartbeat from core client for 30 sec - exiting 19:10:45 (3340): No heartbeat from core client for 30 sec - exiting 19:10:46 (3340): No heartbeat from core client for 30 sec - exiting 19:10:47 (3340): No heartbeat from core client for 30 sec - exiting 19:10:48 (3340): No heartbeat from core client for 30 sec - exiting 19:10:49 (3340): No heartbeat from core client for 30 sec - exiting 19:10:50 (3340): No heartbeat from core client for 30 sec - exiting 19:10:52 (3340): No heartbeat from core client for 30 sec - exiting 19:10:53 (3340): No heartbeat from core client for 30 sec - exiting 19:10:54 (3340): No heartbeat from core client for 30 sec - exiting 19:10:55 (3340): No heartbeat from core client for 30 sec - exiting 19:10:56 (3340): No heartbeat from core client for 30 sec - exiting 19:10:57 (3340): No heartbeat from core client for 30 sec - exiting 19:10:58 (3340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:33:38 (4940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:34:42 (5048): No heartbeat from core client for 30 sec - exiting 18:34:44 (5048): No heartbeat from core client for 30 sec - exiting 18:34:45 (5048): No heartbeat from core client for 30 sec - exiting 18:34:46 (5048): No heartbeat from core client for 30 sec - exiting 18:34:47 (5048): No heartbeat from core client for 30 sec - exiting 18:34:48 (5048): No heartbeat from core client for 30 sec - exiting 18:34:49 (5048): No heartbeat from core client for 30 sec - exiting 18:34:50 (5048): No heartbeat from core client for 30 sec - exiting 18:34:51 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:12:41 (3088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:32 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:51:47 (6328): No heartbeat from core client for 30 sec - exiting 09:51:48 (6328): No heartbeat from core client for 30 sec - exiting 09:51:49 (6328): No heartbeat from core client for 30 sec - exiting 09:51:50 (6328): No heartbeat from core client for 30 sec - exiting 09:51:52 (6328): No heartbeat from core client for 30 sec - exiting 09:51:53 (6328): No heartbeat from core client for 30 sec - exiting 09:51:54 (6328): No heartbeat from core client for 30 sec - exiting 09:51:55 (6328): No heartbeat from core client for 30 sec - exiting 09:51:56 (6328): No heartbeat from core client for 30 sec - exiting 09:51:57 (6328): No heartbeat from core client for 30 sec - exiting 09:51:58 (6328): No heartbeat from core client for 30 sec - exiting 09:51:59 (6328): No heartbeat from core client for 30 sec - exiting 09:52:00 (6328): No heartbeat from core client for 30 sec - exiting 09:52:02 (6328): No heartbeat from core client for 30 sec - exiting 09:52:03 (6328): No heartbeat from core client for 30 sec - exiting 09:52:04 (6328): No heartbeat from core client for 30 sec - exiting 09:52:05 (6328): No heartbeat from core client for 30 sec - exiting 09:52:06 (6328): No heartbeat from core client for 30 sec - exiting 09:52:07 (6328): No heartbeat from core client for 30 sec - exiting 09:52:08 (6328): No heartbeat from core client for 30 sec - exiting 09:52:09 (6328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:06:06 (6452): No heartbeat from core client for 30 sec - exiting 10:06:07 (6452): No heartbeat from core client for 30 sec - exiting 10:06:08 (6452): No heartbeat from core client for 30 sec - exiting 10:06:09 (6452): No heartbeat from core client for 30 sec - exiting 10:06:10 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:46:29 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:20:36 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:20:32 (5464): No heartbeat from core client for 30 sec - exiting 18:20:33 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:34 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:27:35 (6092): No heartbeat from core client for 30 sec - exiting 20:27:36 (6092): No heartbeat from core client for 30 sec - exiting 20:27:37 (6092): No heartbeat from core client for 30 sec - exiting 20:27:38 (6092): No heartbeat from core client for 30 sec - exiting 20:27:39 (6092): No heartbeat from core client for 30 sec - exiting 20:27:41 (6092): No heartbeat from core client for 30 sec - exiting 20:27:42 (6092): No heartbeat from core client for 30 sec - exiting 20:27:43 (6092): No heartbeat from core client for 30 sec - exiting 20:27:44 (6092): No heartbeat from core client for 30 sec - exiting 20:27:45 (6092): No heartbeat from core client for 30 sec - exiting 20:27:46 (6092): No heartbeat from core client for 30 sec - exiting 20:27:47 (6092): No heartbeat from core client for 30 sec - exiting 20:27:48 (6092): No heartbeat from core client for 30 sec - exiting 20:27:49 (6092): No heartbeat from core client for 30 sec - exiting 20:27:50 (6092): No heartbeat from core client for 30 sec - exiting 20:27:51 (6092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:37 (692): No heartbeat from core client for 30 sec - exiting 06:57:38 (692): No heartbeat from core client for 30 sec - exiting 06:57:39 (692): No heartbeat from core client for 30 sec - exiting 06:57:40 (692): No heartbeat from core client for 30 sec - exiting 06:57:41 (692): No heartbeat from core client for 30 sec - exiting 06:57:42 (692): No heartbeat from core client for 30 sec - exiting 06:57:43 (692): No heartbeat from core client for 30 sec - exiting 06:57:44 (692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:45 (692): No heartbeat from core client for 30 sec - exiting 07:00:53 (1144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:15:54 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:56:13 (6592): No heartbeat from core client for 30 sec - exiting 03:56:15 (6592): No heartbeat from core client for 30 sec - exiting 03:56:16 (6592): No heartbeat from core client for 30 sec - exiting 03:56:17 (6592): No heartbeat from core client for 30 sec - exiting 03:56:18 (6592): No heartbeat from core client for 30 sec - exiting 03:56:19 (6592): No heartbeat from core client for 30 sec - exiting 03:56:20 (6592): No heartbeat from core client for 30 sec - exiting 03:56:21 (6592): No heartbeat from core client for 30 sec - exiting 03:56:22 (6592): No heartbeat from core client for 30 sec - exiting 03:56:23 (6592): No heartbeat from core client for 30 sec - exiting 03:56:24 (6592): No heartbeat from core client for 30 sec - exiting 03:56:26 (6592): No heartbeat from core client for 30 sec - exiting 03:56:27 (6592): No heartbeat from core client for 30 sec - exiting 03:56:28 (6592): No heartbeat from core client for 30 sec - exiting 03:56:29 (6592): No heartbeat from core client for 30 sec - exiting 03:56:30 (6592): No heartbeat from core client for 30 sec - exiting 03:56:31 (6592): No heartbeat from core client for 30 sec - exiting 03:56:32 (6592): No heartbeat from core client for 30 sec - exiting 03:56:33 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6640, iMonCtr=1 Model crash detected, will try to restart... 05:35:15 (828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2108, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:45:58 (1916): No heartbeat from core client for 30 sec - exiting 19:46:00 (1916): No heartbeat from core client for 30 sec - exiting 19:46:01 (1916): No heartbeat from core client for 30 sec - exiting 19:46:02 (1916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:03 (1916): No heartbeat from core client for 30 sec - exiting 19:46:04 (1916): No heartbeat from core client for 30 sec - exiting 19:46:05 (1916): No heartbeat from core client for 30 sec - exiting 19:46:06 (1916): No heartbeat from core client for 30 sec - exiting 19:46:07 (1916): No heartbeat from core client for 30 sec - exiting 19:46:08 (1916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:03:46 (6104): No heartbeat from core client for 30 sec - exiting 20:03:47 (6104): No heartbeat from core client for 30 sec - exiting 20:03:48 (6104): No heartbeat from core client for 30 sec - exiting 20:03:49 (6104): No heartbeat from core client for 30 sec - exiting 20:03:50 (6104): No heartbeat from core client for 30 sec - exiting 20:03:51 (6104): No heartbeat from core client for 30 sec - exiting 20:03:52 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:40:34 (5696): No heartbeat from core client for 30 sec - exiting 01:40:35 (5696): No heartbeat from core client for 30 sec - exiting 01:40:36 (5696): No heartbeat from core client for 30 sec - exiting 01:40:37 (5696): No heartbeat from core client for 30 sec - exiting 01:40:39 (5696): No heartbeat from core client for 30 sec - exiting 01:40:40 (5696): No heartbeat from core client for 30 sec - exiting 01:40:41 (5696): No heartbeat from core client for 30 sec - exiting 01:40:42 (5696): No heartbeat from core client for 30 sec - exiting 01:40:43 (5696): No heartbeat from core client for 30 sec - exiting 01:40:44 (5696): No heartbeat from core client for 30 sec - exiting 01:40:45 (5696): No heartbeat from core client for 30 sec - exiting 01:40:46 (5696): No heartbeat from core client for 30 sec - exiting 01:40:47 (5696): No heartbeat from core client for 30 sec - exiting 01:40:48 (5696): No heartbeat from core client for 30 sec - exiting 01:40:50 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:47:15 (5272): No heartbeat from core client for 30 sec - exiting 18:47:16 (5272): No heartbeat from core client for 30 sec - exiting 18:47:17 (5272): No heartbeat from core client for 30 sec - exiting 18:47:18 (5272): No heartbeat from core client for 30 sec - exiting 18:47:19 (5272): No heartbeat from core client for 30 sec - exiting 18:47:21 (5272): No heartbeat from core client for 30 sec - exiting 18:47:22 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:51:35 (3244): No heartbeat from core client for 30 sec - exiting 05:51:36 (3244): No heartbeat from core client for 30 sec - exiting 05:51:37 (3244): No heartbeat from core client for 30 sec - exiting 05:51:38 (3244): No heartbeat from core client for 30 sec - exiting 05:51:40 (3244): No heartbeat from core client for 30 sec - exiting 05:51:41 (3244): No heartbeat from core client for 30 sec - exiting 05:51:42 (3244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:09:54 (972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:47 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n35wko.pj96c10 Error converting file to netcdf: dataout/n35wko.pi96c10 Error converting file to netcdf: dataout/n35wko.pf96c10 Error converting file to netcdf: dataout/n35wka.ph96c10 Error converting file to netcdf: dataout/n35wka.pg96c10 Error converting file to netcdf: dataout/n35wka.pe96c10 Error converting file to netcdf: dataout/n35wka.pd96c10 20:20:28 (4568): No heartbeat from core client for 30 sec - exiting 20:20:29 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/n35wko.pj96c10 Error converting file to netcdf: dataout/n35wko.pi96c10 Error converting file to netcdf: dataout/n35wko.pf96c10 Error converting file to netcdf: dataout/n35wka.ph96c10 Error converting file to netcdf: dataout/n35wka.pg96c10 Error converting file to netcdf: dataout/n35wka.pe96c10 Error converting file to netcdf: dataout/n35wka.pd96c10 CPDN Monitor - Quit request from BOINC... 07:26:46 (4704): No heartbeat from core client for 30 sec - exiting 07:26:47 (4704): No heartbeat from core client for 30 sec - exiting 07:26:48 (4704): No heartbeat from core client for 30 sec - exiting 07:26:49 (4704): No heartbeat from core client for 30 sec - exiting 07:26:50 (4704): No heartbeat from core client for 30 sec - exiting 07:26:52 (4704): No heartbeat from core client for 30 sec - exiting 07:26:53 (4704): No heartbeat from core client for 30 sec - exiting 07:26:54 (4704): No heartbeat from core client for 30 sec - exiting 07:26:55 (4704): No heartbeat from core client for 30 sec - exiting 07:26:56 (4704): No heartbeat from core client for 30 sec - exiting 07:26:57 (4704): No heartbeat from core client for 30 sec - exiting 07:26:58 (4704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:33:39 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:52:06 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:42 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:42:51 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:02:22 (5996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:35:49 (5168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:08:42 (7052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:41:10 (940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:34:08 (2016): No heartbeat from core client for 30 sec - exiting 15:34:09 (2016): No heartbeat from core client for 30 sec - exiting 15:34:11 (2016): No heartbeat from core client for 30 sec - exiting 15:34:12 (2016): No heartbeat from core client for 30 sec - exiting 15:34:13 (2016): No heartbeat from core client for 30 sec - exiting 15:34:14 (2016): No heartbeat from core client for 30 sec - exiting 15:34:15 (2016): No heartbeat from core client for 30 sec - exiting 15:34:16 (2016): No heartbeat from core client for 30 sec - exiting 15:34:17 (2016): No heartbeat from core client for 30 sec - exiting 15:34:18 (2016): No heartbeat from core client for 30 sec - exiting 15:34:19 (2016): No heartbeat from core client for 30 sec - exiting 15:34:20 (2016): No heartbeat from core client for 30 sec - exiting 15:34:21 (2016): No heartbeat from core client for 30 sec - exiting 15:34:23 (2016): No heartbeat from core client for 30 sec - exiting 15:34:24 (2016): No heartbeat from core client for 30 sec - exiting 15:34:25 (2016): No heartbeat from core client for 30 sec - exiting 15:34:26 (2016): No heartbeat from core client for 30 sec - exiting 15:34:27 (2016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:33 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:13:07 (724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6568, iMonCtr=1 Model crash detected, will try to restart... 07:02:50 (5856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:16 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:22 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:26:58 (5672): No heartbeat from core client for 30 sec - exiting 07:26:59 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:53 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:29 (1048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:41:37 (6352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:42:19 (6832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:51:25 (5480): No heartbeat from core client for 30 sec - exiting 12:51:26 (5480): No heartbeat from core client for 30 sec - exiting 12:51:27 (5480): No heartbeat from core client for 30 sec - exiting 12:51:28 (5480): No heartbeat from core client for 30 sec - exiting 12:51:29 (5480): No heartbeat from core client for 30 sec - exiting 12:51:30 (5480): No heartbeat from core client for 30 sec - exiting 12:51:32 (5480): No heartbeat from core client for 30 sec - exiting 12:51:33 (5480): No heartbeat from core client for 30 sec - exiting 12:51:34 (5480): No heartbeat from core client for 30 sec - exiting 12:51:35 (5480): No heartbeat from core client for 30 sec - exiting 12:51:36 (5480): No heartbeat from core client for 30 sec - exiting 12:51:37 (5480): No heartbeat from core client for 30 sec - exiting 12:51:38 (5480): No heartbeat from core client for 30 sec - exiting 12:51:39 (5480): No heartbeat from core client for 30 sec - exiting 12:51:40 (5480): No heartbeat from core client for 30 sec - exiting 12:51:41 (5480): No heartbeat from core client for 30 sec - exiting 12:51:42 (5480): No heartbeat from core client for 30 sec - exiting 12:51:44 (5480): No heartbeat from core client for 30 sec - exiting 12:51:45 (5480): No heartbeat from core client for 30 sec - exiting 12:51:46 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:51:47 (5480): No heartbeat from core client for 30 sec - exiting 12:53:21 (4984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:54:23 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7700, iMonCtr=1 Model crash detected, will try to restart... 06:01:09 (5744): No heartbeat from core client for 30 sec - exiting 06:01:10 (5744): No heartbeat from core client for 30 sec - exiting 06:01:11 (5744): No heartbeat from core client for 30 sec - exiting 06:01:12 (5744): No heartbeat from core client for 30 sec - exiting 06:01:13 (5744): No heartbeat from core client for 30 sec - exiting 06:01:14 (5744): No heartbeat from core client for 30 sec - exiting 06:01:16 (5744): No heartbeat from core client for 30 sec - exiting 06:01:17 (5744): No heartbeat from core client for 30 sec - exiting 06:01:18 (5744): No heartbeat from core client for 30 sec - exiting 06:01:19 (5744): No heartbeat from core client for 30 sec - exiting 06:01:20 (5744): No heartbeat from core client for 30 sec - exiting 06:01:21 (5744): No heartbeat from core client for 30 sec - exiting 06:01:22 (5744): No heartbeat from core client for 30 sec - exiting 06:01:23 (5744): No heartbeat from core client for 30 sec - exiting 06:01:24 (5744): No heartbeat from core client for 30 sec - exiting 06:01:25 (5744): No heartbeat from core client for 30 sec - exiting 06:01:27 (5744): No heartbeat from core client for 30 sec - exiting 06:01:28 (5744): No heartbeat from core client for 30 sec - exiting 06:01:29 (5744): No heartbeat from core client for 30 sec - exiting 06:01:30 (5744): No heartbeat from core client for 30 sec - exiting 06:01:31 (5744): No heartbeat from core client for 30 sec - exiting 06:01:32 (5744): No heartbeat from core client for 30 sec - exiting 06:01:33 (5744): No heartbeat from core client for 30 sec - exiting 06:01:34 (5744): No heartbeat from core client for 30 sec - exiting 06:01:35 (5744): No heartbeat from core client for 30 sec - exiting 06:01:36 (5744): No heartbeat from core client for 30 sec - exiting 06:01:37 (5744): No heartbeat from core client for 30 sec - exiting 06:01:39 (5744): No heartbeat from core client for 30 sec - exiting 06:01:40 (5744): No heartbeat from core client for 30 sec - exiting 06:01:41 (5744): No heartbeat from core client for 30 sec - exiting 06:01:42 (5744): No heartbeat from core client for 30 sec - exiting 06:01:43 (5744): No heartbeat from core client for 30 sec - exiting 06:01:44 (5744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:31 (2408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:28:59 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:45:34 (5300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:46:30 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:49:50 (6904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:30:47 (1784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:33:15 (260): No heartbeat from core client for 30 sec - exiting 06:33:16 (260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:20 (3480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:56:12 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:57:15 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:16 (4412): No heartbeat from core client for 30 sec - exiting 05:46:17 (4412): No heartbeat from core client for 30 sec - exiting 05:46:18 (4412): No heartbeat from core client for 30 sec - exiting 05:46:19 (4412): No heartbeat from core client for 30 sec - exiting 05:46:21 (4412): No heartbeat from core client for 30 sec - exiting 05:46:22 (4412): No heartbeat from core client for 30 sec - exiting 05:46:23 (4412): No heartbeat from core client for 30 sec - exiting 05:46:24 (4412): No heartbeat from core client for 30 sec - exiting 05:46:25 (4412): No heartbeat from core client for 30 sec - exiting 05:46:26 (4412): No heartbeat from core client for 30 sec - exiting 05:46:27 (4412): No heartbeat from core client for 30 sec - exiting 05:46:28 (4412): No heartbeat from core client for 30 sec - exiting 05:46:29 (4412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:30 (4412): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:01:40 (6884): No heartbeat from core client for 30 sec - exiting 21:01:41 (6884): No heartbeat from core client for 30 sec - exiting 21:01:43 (6884): No heartbeat from core client for 30 sec - exiting 21:01:44 (6884): No heartbeat from core client for 30 sec - exiting 21:01:45 (6884): No heartbeat from core client for 30 sec - exiting 21:01:46 (6884): No heartbeat from core client for 30 sec - exiting 21:01:47 (6884): No heartbeat from core client for 30 sec - exiting 21:01:49 (6884): No heartbeat from core client for 30 sec - exiting 21:01:50 (6884): No heartbeat from core client for 30 sec - exiting 21:01:51 (6884): No heartbeat from core client for 30 sec - exiting 21:01:52 (6884): No heartbeat from core client for 30 sec - exiting 21:01:53 (6884): No heartbeat from core client for 30 sec - exiting 21:01:54 (6884): No heartbeat from core client for 30 sec - exiting 21:01:55 (6884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 02:55:02 (4208): No heartbeat from core client for 30 sec - exiting 02:55:03 (4208): No heartbeat from core client for 30 sec - exiting 02:55:04 (4208): No heartbeat from core client for 30 sec - exiting 02:55:05 (4208): No heartbeat from core client for 30 sec - exiting 02:55:06 (4208): No heartbeat from core client for 30 sec - exiting 02:55:07 (4208): No heartbeat from core client for 30 sec - exiting 02:55:08 (4208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:08:45 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:03 (6248): No heartbeat from core client for 30 sec - exiting 06:41:04 (6248): No heartbeat from core client for 30 sec - exiting 06:41:05 (6248): No heartbeat from core client for 30 sec - exiting 06:41:06 (6248): No heartbeat from core client for 30 sec - exiting 06:41:07 (6248): No heartbeat from core client for 30 sec - exiting 06:41:08 (6248): No heartbeat from core client for 30 sec - exiting 06:41:10 (6248): No heartbeat from core client for 30 sec - exiting 06:41:11 (6248): No heartbeat from core client for 30 sec - exiting 06:41:12 (6248): No heartbeat from core client for 30 sec - exiting 06:41:13 (6248): No heartbeat from core client for 30 sec - exiting 06:41:14 (6248): No heartbeat from core client for 30 sec - exiting 06:41:15 (6248): No heartbeat from core client for 30 sec - exiting 06:41:16 (6248): No heartbeat from core client for 30 sec - exiting 06:41:17 (6248): No heartbeat from core client for 30 sec - exiting 06:41:18 (6248): No heartbeat from core client for 30 sec - exiting 06:41:19 (6248): No heartbeat from core client for 30 sec - exiting 06:41:21 (6248): No heartbeat from core client for 30 sec - exiting 06:41:22 (6248): No heartbeat from core client for 30 sec - exiting 06:41:23 (6248): No heartbeat from core client for 30 sec - exiting 06:41:24 (6248): No heartbeat from core client for 30 sec - exiting 06:41:25 (6248): No heartbeat from core client for 30 sec - exiting 06:41:26 (6248): No heartbeat from core client for 30 sec - exiting 06:41:27 (6248): No heartbeat from core client for 30 sec - exiting 06:41:28 (6248): No heartbeat from core client for 30 sec - exiting 06:41:29 (6248): No heartbeat from core client for 30 sec - exiting 06:41:30 (6248): No heartbeat from core client for 30 sec - exiting 06:41:31 (6248): No heartbeat from core client for 30 sec - exiting 06:41:32 (6248): No heartbeat from core client for 30 sec - exiting 06:41:34 (6248): No heartbeat from core client for 30 sec - exiting 06:41:35 (6248): No heartbeat from core client for 30 sec - exiting 06:41:36 (6248): No heartbeat from core client for 30 sec - exiting 06:41:37 (6248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:30 (5500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:04 (6212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3876, iMonCtr=1 Model crash detected, will try to restart... 17:33:15 (2416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:37 (1724): No heartbeat from core client for 30 sec - exiting 20:24:38 (1724): No heartbeat from core client for 30 sec - exiting 20:24:40 (1724): No heartbeat from core client for 30 sec - exiting 20:24:41 (1724): No heartbeat from core client for 30 sec - exiting 20:24:42 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:04 (6968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:43:02 (564): No heartbeat from core client for 30 sec - exiting 05:43:03 (564): No heartbeat from core client for 30 sec - exiting 05:43:04 (564): No heartbeat from core client for 30 sec - exiting 05:43:05 (564): No heartbeat from core client for 30 sec - exiting 05:43:06 (564): No heartbeat from core client for 30 sec - exiting 05:43:07 (564): No heartbeat from core client for 30 sec - exiting 05:43:08 (564): No heartbeat from core client for 30 sec - exiting 05:43:09 (564): No heartbeat from core client for 30 sec - exiting 05:43:10 (564): No heartbeat from core client for 30 sec - exiting 05:43:12 (564): No heartbeat from core client for 30 sec - exiting 05:43:13 (564): No heartbeat from core client for 30 sec - exiting 05:43:14 (564): No heartbeat from core client for 30 sec - exiting 05:43:15 (564): No heartbeat from core client for 30 sec - exiting 05:43:16 (564): No heartbeat from core client for 30 sec - exiting 05:43:17 (564): No heartbeat from core client for 30 sec - exiting 05:43:19 (564): No heartbeat from core client for 30 sec - exiting 05:43:20 (564): No heartbeat from core client for 30 sec - exiting 05:43:21 (564): No heartbeat from core client for 30 sec - exiting 05:43:22 (564): No heartbeat from core client for 30 sec - exiting 05:43:23 (564): No heartbeat from core client for 30 sec - exiting 05:43:24 (564): No heartbeat from core client for 30 sec - exiting 05:43:25 (564): No heartbeat from core client for 30 sec - exiting 05:43:26 (564): No heartbeat from core client for 30 sec - exiting 05:43:27 (564): No heartbeat from core client for 30 sec - exiting 05:43:28 (564): No heartbeat from core client for 30 sec - exiting 05:43:30 (564): No heartbeat from core client for 30 sec - exiting 05:43:31 (564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:17 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:53 (5664): No heartbeat from core client for 30 sec - exiting 19:55:55 (5664): No heartbeat from core client for 30 sec - exiting 19:55:56 (5664): No heartbeat from core client for 30 sec - exiting 19:55:57 (5664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2560, iMonCtr=1 Model crash detected, will try to restart... 19:18:05 (6336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:06:54 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6768, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6768, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6768, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6848, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6848, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Oct 2013 11:05:56	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	777,600	1,059,914	1.3631
14 Oct 2013 03:24:37	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	751,680	1,021,144	1.3585
13 Oct 2013 05:43:11	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	725,760	982,706	1.3540
12 Oct 2013 16:56:57	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	699,840	944,157	1.3491
12 Oct 2013 02:57:34	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	673,920	903,851	1.3412
10 Oct 2013 12:43:33	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	648,000	863,142	1.3320
06 Oct 2013 04:12:09	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	622,080	822,090	1.3215
05 Oct 2013 06:08:00	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	596,160	781,294	1.3105
03 Oct 2013 13:03:36	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	570,240	741,801	1.3009
30 Sep 2013 11:26:23	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	544,320	704,253	1.2938
29 Sep 2013 01:26:28	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	518,400	668,136	1.2888
28 Sep 2013 02:06:52	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	492,480	632,845	1.2850
25 Sep 2013 21:07:33	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	466,560	597,540	1.2807
23 Sep 2013 21:03:21	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	440,640	562,332	1.2762
23 Sep 2013 11:37:28	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	414,720	525,645	1.2675
21 Sep 2013 14:07:18	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	388,800	489,838	1.2599
20 Sep 2013 14:57:48	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	362,880	453,883	1.2508
18 Sep 2013 14:29:00	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	336,960	417,060	1.2377
16 Sep 2013 13:03:13	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	311,040	381,004	1.2249
15 Sep 2013 16:06:13	1288622	15997441	hadcm3n_n35w_1880_40_008374073_1	285,120	346,591	1.2156