Task 13500098

Name	hadcm3n_p78x_1940_40_007421653_2
Workunit	7619288
Created	16 Oct 2011, 8:03:43 UTC
Sent	16 Oct 2011, 8:16:15 UTC
Report deadline	15 Jan 2012, 15:43:26 UTC
Received	6 Nov 2011, 1:42:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1175495
Run time	5 days 21 hours 25 min 13 sec
CPU time	5 days 4 hours 27 min 37 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.38 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> 09:55:59 (5168): No heartbeat from core client for 30 sec - exiting 09:56:00 (5168): No heartbeat from core client for 30 sec - exiting 09:56:01 (5168): No heartbeat from core client for 30 sec - exiting 09:56:02 (5168): No heartbeat from core client for 30 sec - exiting 09:56:03 (5168): No heartbeat from core client for 30 sec - exiting 09:56:04 (5168): No heartbeat from core client for 30 sec - exiting 09:56:05 (5168): No heartbeat from core client for 30 sec - exiting 09:56:06 (5168): No heartbeat from core client for 30 sec - exiting 09:56:07 (5168): No heartbeat from core client for 30 sec - exiting 09:56:09 (5168): No heartbeat from core client for 30 sec - exiting 09:56:10 (5168): No heartbeat from core client for 30 sec - exiting 09:56:11 (5168): No heartbeat from core client for 30 sec - exiting 09:56:12 (5168): No heartbeat from core client for 30 sec - exiting 09:56:13 (5168): No heartbeat from core client for 30 sec - exiting 09:56:14 (5168): No heartbeat from core client for 30 sec - exiting 09:56:15 (5168): No heartbeat from core client for 30 sec - exiting 09:56:16 (5168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:16 (196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:00:58 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5368, iMonCtr=1 Model crash detected, will try to restart... 11:03:53 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:50:29 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:15:24 (5392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:47:34 (5048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4556, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:58:49 (4976): No heartbeat from core client for 30 sec - exiting 08:58:50 (4976): No heartbeat from core client for 30 sec - exiting 08:58:51 (4976): No heartbeat from core client for 30 sec - exiting 08:58:52 (4976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pje1c10 Error converting file to netcdf: dataout/p78xko.pie1c10 Error converting file to netcdf: dataout/p78xko.pfe1c10 Error converting file to netcdf: dataout/p78xka.phe1c10 Error converting file to netcdf: dataout/p78xka.pge1c10 Error converting file to netcdf: dataout/p78xka.pee1c10 Error converting file to netcdf: dataout/p78xka.pde1c10 MainError: 06:34:45 AM No files match the supplied pattern. MainError: 06:34:45 AM No files match the supplied pattern. 17:11:31 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/p78xko.pje2c10 Error converting file to netcdf: dataout/p78xko.pie2c10 Error converting file to netcdf: dataout/p78xko.pfe2c10 Error converting file to netcdf: dataout/p78xka.phe2c10 Error converting file to netcdf: dataout/p78xka.pge2c10 Error converting file to netcdf: dataout/p78xka.pee2c10 Error converting file to netcdf: dataout/p78xka.pde2c10 MainError: 01:33:08 PM No files match the supplied pattern. MainError: 01:33:08 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:58:51 (760): No heartbeat from core client for 30 sec - exiting 14:58:52 (760): No heartbeat from core client for 30 sec - exiting 14:58:53 (760): No heartbeat from core client for 30 sec - exiting 14:58:54 (760): No heartbeat from core client for 30 sec - exiting 14:58:55 (760): No heartbeat from core client for 30 sec - exiting 14:58:56 (760): No heartbeat from core client for 30 sec - exiting 14:58:58 (760): No heartbeat from core client for 30 sec - exiting 14:58:59 (760): No heartbeat from core client for 30 sec - exiting 14:59:00 (760): No heartbeat from core client for 30 sec - exiting 14:59:01 (760): No heartbeat from core client for 30 sec - exiting 14:59:02 (760): No heartbeat from core client for 30 sec - exiting 14:59:03 (760): No heartbeat from core client for 30 sec - exiting 14:59:04 (760): No heartbeat from core client for 30 sec - exiting 14:59:05 (760): No heartbeat from core client for 30 sec - exiting 14:59:06 (760): No heartbeat from core client for 30 sec - exiting 14:59:07 (760): No heartbeat from core client for 30 sec - exiting 14:59:09 (760): No heartbeat from core client for 30 sec - exiting 14:59:10 (760): No heartbeat from core client for 30 sec - exiting 14:59:11 (760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:49 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:13:01 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:02 (5268): No heartbeat from core client for 30 sec - exiting 09:13:03 (5268): No heartbeat from core client for 30 sec - exiting Error converting file to netcdf: dataout/p78xko.pje3c10 Error converting file to netcdf: dataout/p78xko.pie3c10 Error converting file to netcdf: dataout/p78xko.pfe3c10 Error converting file to netcdf: dataout/p78xka.phe3c10 Error converting file to netcdf: dataout/p78xka.pge3c10 Error converting file to netcdf: dataout/p78xka.pee3c10 Error converting file to netcdf: dataout/p78xka.pde3c10 MainError: 03:44:11 AM No files match the supplied pattern. MainError: 03:44:11 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pje4c10 Error converting file to netcdf: dataout/p78xko.pie4c10 Error converting file to netcdf: dataout/p78xko.pfe4c10 Error converting file to netcdf: dataout/p78xka.phe4c10 Error converting file to netcdf: dataout/p78xka.pge4c10 Error converting file to netcdf: dataout/p78xka.pee4c10 Error converting file to netcdf: dataout/p78xka.pde4c10 MainError: 11:49:14 AM No files match the supplied pattern. MainError: 11:49:14 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:38 (5504): No heartbeat from core client for 30 sec - exiting 21:25:40 (5504): No heartbeat from core client for 30 sec - exiting 21:25:41 (5504): No heartbeat from core client for 30 sec - exiting 21:25:42 (5504): No heartbeat from core client for 30 sec - exiting 21:25:43 (5504): No heartbeat from core client for 30 sec - exiting 21:25:44 (5504): No heartbeat from core client for 30 sec - exiting 21:25:45 (5504): No heartbeat from core client for 30 sec - exiting 21:25:46 (5504): No heartbeat from core client for 30 sec - exiting 21:25:47 (5504): No heartbeat from core client for 30 sec - exiting 21:25:48 (5504): No heartbeat from core client for 30 sec - exiting 21:25:50 (5504): No heartbeat from core client for 30 sec - exiting 21:25:51 (5504): No heartbeat from core client for 30 sec - exiting 21:25:52 (5504): No heartbeat from core client for 30 sec - exiting 21:25:53 (5504): No heartbeat from core client for 30 sec - exiting 21:25:54 (5504): No heartbeat from core client for 30 sec - exiting 21:25:55 (5504): No heartbeat from core client for 30 sec - exiting 21:25:56 (5504): No heartbeat from core client for 30 sec - exiting 21:25:57 (5504): No heartbeat from core client for 30 sec - exiting 21:25:58 (5504): No heartbeat from core client for 30 sec - exiting 21:25:59 (5504): No heartbeat from core client for 30 sec - exiting 21:26:01 (5504): No heartbeat from core client for 30 sec - exiting 21:26:02 (5504): No heartbeat from core client for 30 sec - exiting 21:26:03 (5504): No heartbeat from core client for 30 sec - exiting 21:26:04 (5504): No heartbeat from core client for 30 sec - exiting 21:26:05 (5504): No heartbeat from core client for 30 sec - exiting 21:26:06 (5504): No heartbeat from core client for 30 sec - exiting 21:26:07 (5504): No heartbeat from core client for 30 sec - exiting 21:26:08 (5504): No heartbeat from core client for 30 sec - exiting 21:26:09 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:48 (5056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:59 (4752): No heartbeat from core client for 30 sec - exiting 07:40:00 (4752): No heartbeat from core client for 30 sec - exiting 07:40:01 (4752): No heartbeat from core client for 30 sec - exiting 07:40:02 (4752): No heartbeat from core client for 30 sec - exiting 07:40:03 (4752): No heartbeat from core client for 30 sec - exiting 07:40:04 (4752): No heartbeat from core client for 30 sec - exiting 07:40:05 (4752): No heartbeat from core client for 30 sec - exiting 07:40:06 (4752): No heartbeat from core client for 30 sec - exiting 07:40:08 (4752): No heartbeat from core client for 30 sec - exiting 07:40:09 (4752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/p78xko.pje5c10 Error converting file to netcdf: dataout/p78xko.pie5c10 Error converting file to netcdf: dataout/p78xko.pfe5c10 Error converting file to netcdf: dataout/p78xka.phe5c10 Error converting file to netcdf: dataout/p78xka.pge5c10 Error converting file to netcdf: dataout/p78xka.pee5c10 Error converting file to netcdf: dataout/p78xka.pde5c10 MainError: 12:11:33 AM No files match the supplied pattern. MainError: 12:11:33 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:40:30 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:05 (6732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/p78xko.pje6c10 Error converting file to netcdf: dataout/p78xko.pie6c10 Error converting file to netcdf: dataout/p78xko.pfe6c10 Error converting file to netcdf: dataout/p78xka.phe6c10 Error converting file to netcdf: dataout/p78xka.pge6c10 Error converting file to netcdf: dataout/p78xka.pee6c10 Error converting file to netcdf: dataout/p78xka.pde6c10 MainError: 08:48:50 AM No files match the supplied pattern. MainError: 08:48:50 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pje7c10 Error converting file to netcdf: dataout/p78xko.pie7c10 Error converting file to netcdf: dataout/p78xko.pfe7c10 Error converting file to netcdf: dataout/p78xka.phe7c10 Error converting file to netcdf: dataout/p78xka.pge7c10 Error converting file to netcdf: dataout/p78xka.pee7c10 Error converting file to netcdf: dataout/p78xka.pde7c10 MainError: 05:00:48 AM No files match the supplied pattern. MainError: 05:00:48 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pje8c10 Error converting file to netcdf: dataout/p78xko.pie8c10 Error converting file to netcdf: dataout/p78xko.pfe8c10 Error converting file to netcdf: dataout/p78xka.phe8c10 Error converting file to netcdf: dataout/p78xka.pge8c10 Error converting file to netcdf: dataout/p78xka.pee8c10 Error converting file to netcdf: dataout/p78xka.pde8c10 MainError: 01:07:04 PM No files match the supplied pattern. MainError: 01:07:04 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5964, iMonCtr=1 Model crash detected, will try to restart... 10:14:48 (5064): No heartbeat from core client for 30 sec - exiting 10:14:50 (5064): No heartbeat from core client for 30 sec - exiting 10:14:51 (5064): No heartbeat from core client for 30 sec - exiting 10:14:52 (5064): No heartbeat from core client for 30 sec - exiting 10:14:53 (5064): No heartbeat from core client for 30 sec - exiting 10:14:54 (5064): No heartbeat from core client for 30 sec - exiting 10:14:55 (5064): No heartbeat from core client for 30 sec - exiting 10:14:56 (5064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:54 (2740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:47:55 (2452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/p78xko.pje9c10 Error converting file to netcdf: dataout/p78xko.pie9c10 Error converting file to netcdf: dataout/p78xko.pfe9c10 Error converting file to netcdf: dataout/p78xka.phe9c10 Error converting file to netcdf: dataout/p78xka.pge9c10 Error converting file to netcdf: dataout/p78xka.pee9c10 Error converting file to netcdf: dataout/p78xka.pde9c10 MainError: 07:08:16 AM No files match the supplied pattern. MainError: 07:08:16 AM No files match the supplied pattern. 14:31:48 (5060): No heartbeat from core client for 30 sec - exiting 14:31:49 (5060): No heartbeat from core client for 30 sec - exiting 14:31:50 (5060): No heartbeat from core client for 30 sec - exiting 14:31:51 (5060): No heartbeat from core client for 30 sec - exiting 14:31:53 (5060): No heartbeat from core client for 30 sec - exiting 14:31:54 (5060): No heartbeat from core client for 30 sec - exiting 14:31:55 (5060): No heartbeat from core client for 30 sec - exiting 14:31:56 (5060): No heartbeat from core client for 30 sec - exiting 14:31:57 (5060): No heartbeat from core client for 30 sec - exiting 14:31:58 (5060): No heartbeat from core client for 30 sec - exiting 14:31:59 (5060): No heartbeat from core client for 30 sec - exiting 14:32:00 (5060): No heartbeat from core client for 30 sec - exiting 14:32:01 (5060): No heartbeat from core client for 30 sec - exiting 14:32:02 (5060): No heartbeat from core client for 30 sec - exiting 14:32:03 (5060): No heartbeat from core client for 30 sec - exiting 14:32:05 (5060): No heartbeat from core client for 30 sec - exiting 14:32:06 (5060): No heartbeat from core client for 30 sec - exiting 14:32:07 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pjf0c10 Error converting file to netcdf: dataout/p78xko.pif0c10 Error converting file to netcdf: dataout/p78xko.pff0c10 Error converting file to netcdf: dataout/p78xko.pcf0c10 Error converting file to netcdf: dataout/p78xko.pbf0c10 Error converting file to netcdf: dataout/p78xko.paf0c10 Error converting file to netcdf: dataout/p78xka.phf0c10 Error converting file to netcdf: dataout/p78xka.pgf0c10 Error converting file to netcdf: dataout/p78xka.pef0c10 Error converting file to netcdf: dataout/p78xka.pdf0c10 MainError: 02:35:08 PM No files match the supplied pattern. MainError: 02:35:08 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:31:29 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/p78xko.pjf1c10 Error converting file to netcdf: dataout/p78xko.pif1c10 Error converting file to netcdf: dataout/p78xko.pff1c10 Error converting file to netcdf: dataout/p78xka.phf1c10 Error converting file to netcdf: dataout/p78xka.pgf1c10 Error converting file to netcdf: dataout/p78xka.pef1c10 Error converting file to netcdf: dataout/p78xka.pdf1c10 MainError: 12:27:37 AM No files match the supplied pattern. MainError: 12:27:37 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/p78xko.pjf2c10 Error converting file to netcdf: dataout/p78xko.pif2c10 Error converting file to netcdf: dataout/p78xko.pff2c10 Error converting file to netcdf: dataout/p78xka.phf2c10 Error converting file to netcdf: dataout/p78xka.pgf2c10 Error converting file to netcdf: dataout/p78xka.pef2c10 Error converting file to netcdf: dataout/p78xka.pdf2c10 MainError: 09:41:02 AM No files match the supplied pattern. MainError: 09:41:02 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6560, iMonCtr=1 Model crash detected, will try to restart... 07:16:06 (4972): No heartbeat from core client for 30 sec - exiting 07:16:07 (4972): No heartbeat from core client for 30 sec - exiting 07:16:08 (4972): No heartbeat from core client for 30 sec - exiting 07:16:09 (4972): No heartbeat from core client for 30 sec - exiting 07:16:11 (4972): No heartbeat from core client for 30 sec - exiting 07:16:12 (4972): No heartbeat from core client for 30 sec - exiting 07:16:13 (4972): No heartbeat from core client for 30 sec - exiting 07:16:14 (4972): No heartbeat from core client for 30 sec - exiting 07:16:15 (4972): No heartbeat from core client for 30 sec - exiting 07:16:16 (4972): No heartbeat from core client for 30 sec - exiting 07:16:17 (4972): No heartbeat from core client for 30 sec - exiting 07:16:18 (4972): No heartbeat from core client for 30 sec - exiting 07:16:19 (4972): No heartbeat from core client for 30 sec - exiting 07:16:20 (4972): No heartbeat from core client for 30 sec - exiting 07:16:21 (4972): No heartbeat from core client for 30 sec - exiting 07:16:23 (4972): No heartbeat from core client for 30 sec - exiting 07:16:24 (4972): No heartbeat from core client for 30 sec - exiting 07:16:25 (4972): No heartbeat from core client for 30 sec - exiting 07:16:26 (4972): No heartbeat from core client for 30 sec - exiting 07:16:27 (4972): No heartbeat from core client for 30 sec - exiting 07:16:28 (4972): No heartbeat from core client for 30 sec - exiting 07:16:29 (4972): No heartbeat from core client for 30 sec - exiting 07:16:30 (4972): No heartbeat from core client for 30 sec - exiting 07:16:31 (4972): No heartbeat from core client for 30 sec - exiting 07:16:32 (4972): No heartbeat from core client for 30 sec - exiting 07:16:33 (4972): No heartbeat from core client for 30 sec - exiting 07:16:35 (4972): No heartbeat from core client for 30 sec - exiting 07:16:36 (4972): No heartbeat from core client for 30 sec - exiting 07:16:37 (4972): No heartbeat from core client for 30 sec - exiting 07:16:38 (4972): No heartbeat from core client for 30 sec - exiting 07:16:39 (4972): No heartbeat from core client for 30 sec - exiting 07:16:40 (4972): No heartbeat from core client for 30 sec - exiting 07:16:41 (4972): No heartbeat from core client for 30 sec - exiting 07:16:42 (4972): No heartbeat from core client for 30 sec - exiting 07:16:43 (4972): No heartbeat from core client for 30 sec - exiting 07:16:44 (4972): No heartbeat from core client for 30 sec - exiting 07:16:45 (4972): No heartbeat from core client for 30 sec - exiting 07:16:47 (4972): No heartbeat from core client for 30 sec - exiting 07:16:48 (4972): No heartbeat from core client for 30 sec - exiting 07:16:49 (4972): No heartbeat from core client for 30 sec - exiting 07:16:50 (4972): No heartbeat from core client for 30 sec - exiting 07:16:51 (4972): No heartbeat from core client for 30 sec - exiting 07:16:52 (4972): No heartbeat from core client for 30 sec - exiting 07:16:53 (4972): No heartbeat from core client for 30 sec - exiting 07:16:54 (4972): No heartbeat from core client for 30 sec - exiting 07:16:55 (4972): No heartbeat from core client for 30 sec - exiting 07:16:56 (4972): No heartbeat from core client for 30 sec - exiting 07:16:57 (4972): No heartbeat from core client for 30 sec - exiting 07:16:59 (4972): No heartbeat from core client for 30 sec - exiting 07:17:00 (4972): No heartbeat from core client for 30 sec - exiting 07:17:01 (4972): No heartbeat from core client for 30 sec - exiting 07:17:02 (4972): No heartbeat from core client for 30 sec - exiting 07:17:03 (4972): No heartbeat from core client for 30 sec - exiting 07:17:04 (4972): No heartbeat from core client for 30 sec - exiting 07:17:05 (4972): No heartbeat from core client for 30 sec - exiting 07:17:06 (4972): No heartbeat from core client for 30 sec - exiting 07:17:07 (4972): No heartbeat from core client for 30 sec - exiting 07:17:08 (4972): No heartbeat from core client for 30 sec - exiting 07:17:09 (4972): No heartbeat from core client for 30 sec - exiting 07:17:11 (4972): No heartbeat from core client for 30 sec - exiting 07:17:12 (4972): No heartbeat from core client for 30 sec - exiting 07:17:13 (4972): No heartbeat from core client for 30 sec - exiting 07:17:14 (4972): No heartbeat from core client for 30 sec - exiting 07:17:15 (4972): No heartbeat from core client for 30 sec - exiting 07:17:16 (4972): No heartbeat from core client for 30 sec - exiting 07:17:17 (4972): No heartbeat from core client for 30 sec - exiting 07:17:18 (4972): No heartbeat from core client for 30 sec - exiting 07:17:19 (4972): No heartbeat from core client for 30 sec - exiting 07:17:20 (4972): No heartbeat from core client for 30 sec - exiting 07:17:21 (4972): No heartbeat from core client for 30 sec - exiting 07:17:23 (4972): No heartbeat from core client for 30 sec - exiting 07:17:24 (4972): No heartbeat from core client for 30 sec - exiting 07:17:25 (4972): No heartbeat from core client for 30 sec - exiting 07:17:26 (4972): No heartbeat from core client for 30 sec - exiting 07:17:27 (4972): No heartbeat from core client for 30 sec - exiting 07:17:28 (4972): No heartbeat from core client for 30 sec - exiting 07:17:29 (4972): No heartbeat from core client for 30 sec - exiting 07:17:30 (4972): No heartbeat from core client for 30 sec - exiting 07:17:31 (4972): No heartbeat from core client for 30 sec - exiting 07:17:32 (4972): No heartbeat from core client for 30 sec - exiting 07:17:33 (4972): No heartbeat from core client for 30 sec - exiting 07:17:35 (4972): No heartbeat from core client for 30 sec - exiting 07:17:36 (4972): No heartbeat from core client for 30 sec - exiting 07:17:37 (4972): No heartbeat from core client for 30 sec - exiting 07:17:38 (4972): No heartbeat from core client for 30 sec - exiting 07:17:39 (4972): No heartbeat from core client for 30 sec - exiting 07:17:40 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:22:58 (1120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/p78xko.pjf3c10 Error converting file to netcdf: dataout/p78xko.pif3c10 Error converting file to netcdf: dataout/p78xko.pff3c10 Error converting file to netcdf: dataout/p78xka.phf3c10 Error converting file to netcdf: dataout/p78xka.pgf3c10 Error converting file to netcdf: dataout/p78xka.pef3c10 Error converting file to netcdf: dataout/p78xka.pdf3c10 MainError: 12:11:37 AM No files match the supplied pattern. MainError: 12:11:37 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:41:14 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:26:25 (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7788, iMonCtr=1 Model crash detected, will try to restart... 07:00:12 (4652): No heartbeat from core client for 30 sec - exiting 07:00:14 (4652): No heartbeat from core client for 30 sec - exiting 07:00:15 (4652): No heartbeat from core client for 30 sec - exiting 07:00:16 (4652): No heartbeat from core client for 30 sec - exiting 07:00:17 (4652): No heartbeat from core client for 30 sec - exiting 07:00:18 (4652): No heartbeat from core client for 30 sec - exiting 07:00:19 (4652): No heartbeat from core client for 30 sec - exiting 07:00:20 (4652): No heartbeat from core client for 30 sec - exiting 07:00:21 (4652): No heartbeat from core client for 30 sec - exiting 07:00:22 (4652): No heartbeat from core client for 30 sec - exiting 07:00:23 (4652): No heartbeat from core client for 30 sec - exiting 07:00:24 (4652): No heartbeat from core client for 30 sec - exiting 07:00:26 (4652): No heartbeat from core client for 30 sec - exiting 07:00:27 (4652): No heartbeat from core client for 30 sec - exiting 07:00:28 (4652): No heartbeat from core client for 30 sec - exiting 07:00:29 (4652): No heartbeat from core client for 30 sec - exiting 07:00:30 (4652): No heartbeat from core client for 30 sec - exiting 07:00:31 (4652): No heartbeat from core client for 30 sec - exiting 07:00:32 (4652): No heartbeat from core client for 30 sec - exiting 07:00:33 (4652): No heartbeat from core client for 30 sec - exiting 07:00:34 (4652): No heartbeat from core client for 30 sec - exiting 07:00:35 (4652): No heartbeat from core client for 30 sec - exiting 07:00:36 (4652): No heartbeat from core client for 30 sec - exiting 07:00:38 (4652): No heartbeat from core client for 30 sec - exiting 07:00:39 (4652): No heartbeat from core client for 30 sec - exiting 07:00:40 (4652): No heartbeat from core client for 30 sec - exiting 07:00:41 (4652): No heartbeat from core client for 30 sec - exiting 07:00:42 (4652): No heartbeat from core client for 30 sec - exiting 07:00:43 (4652): No heartbeat from core client for 30 sec - exiting 07:00:44 (4652): No heartbeat from core client for 30 sec - exiting 07:00:45 (4652): No heartbeat from core client for 30 sec - exiting 07:00:46 (4652): No heartbeat from core client for 30 sec - exiting 07:00:47 (4652): No heartbeat from core client for 30 sec - exiting 07:00:48 (4652): No heartbeat from core client for 30 sec - exiting 07:00:50 (4652): No heartbeat from core client for 30 sec - exiting 07:00:51 (4652): No heartbeat from core client for 30 sec - exiting 07:00:52 (4652): No heartbeat from core client for 30 sec - exiting 07:00:53 (4652): No heartbeat from core client for 30 sec - exiting 07:00:54 (4652): No heartbeat from core client for 30 sec - exiting 07:00:55 (4652): No heartbeat from core client for 30 sec - exiting 07:00:56 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:13 (4136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/p78xko.pjf4c10 Error converting file to netcdf: dataout/p78xko.pif4c10 Error converting file to netcdf: dataout/p78xko.pff4c10 Error converting file to netcdf: dataout/p78xka.phf4c10 Error converting file to netcdf: dataout/p78xka.pgf4c10 Error converting file to netcdf: dataout/p78xka.pef4c10 Error converting file to netcdf: dataout/p78xka.pdf4c10 MainError: 11:07:33 PM No files match the supplied pattern. MainError: 11:07:33 PM No files match the supplied pattern. 07:08:08 (3144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 07:08:54 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... 07:20:05 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:57 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:32:19 (4980): No heartbeat from core client for 30 sec - exiting 08:32:20 (4980): No heartbeat from core client for 30 sec - exiting 08:32:21 (4980): No heartbeat from core client for 30 sec - exiting 08:32:22 (4980): No heartbeat from core client for 30 sec - exiting 08:32:23 (4980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:32:24 (4980): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:53:08 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:57:11 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:57:12 (4608): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:11:39 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:34 (4824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:35 (4824): No heartbeat from core client for 30 sec - exiting Error converting file to netcdf: dataout/p78xko.pjf5c10 Error converting file to netcdf: dataout/p78xko.pif5c10 Error converting file to netcdf: dataout/p78xko.pff5c10 Error converting file to netcdf: dataout/p78xka.phf5c10 Error converting file to netcdf: dataout/p78xka.pgf5c10 Error converting file to netcdf: dataout/p78xka.pef5c10 Error converting file to netcdf: dataout/p78xka.pdf5c10 MainError: 07:20:59 AM No files match the supplied pattern. MainError: 07:20:59 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:54:44 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:56:42 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:35 (2196): No heartbeat from core client for 30 sec - exiting 08:18:36 (2196): No heartbeat from core client for 30 sec - exiting 08:18:37 (2196): No heartbeat from core client for 30 sec - exiting 08:18:38 (2196): No heartbeat from core client for 30 sec - exiting 08:18:39 (2196): No heartbeat from core client for 30 sec - exiting 08:18:41 (2196): No heartbeat from core client for 30 sec - exiting 08:18:42 (2196): No heartbeat from core client for 30 sec - exiting 08:18:43 (2196): No heartbeat from core client for 30 sec - exiting 08:18:44 (2196): No heartbeat from core client for 30 sec - exiting 08:18:45 (2196): No heartbeat from core client for 30 sec - exiting 08:18:46 (2196): No heartbeat from core client for 30 sec - exiting 08:18:47 (2196): No heartbeat from core client for 30 sec - exiting 08:18:48 (2196): No heartbeat from core client for 30 sec - exiting 08:18:49 (2196): No heartbeat from core client for 30 sec - exiting 08:18:50 (2196): No heartbeat from core client for 30 sec - exiting 08:18:51 (2196): No heartbeat from core client for 30 sec - exiting 08:18:53 (2196): No heartbeat from core client for 30 sec - exiting 08:18:54 (2196): No heartbeat from core client for 30 sec - exiting 08:18:55 (2196): No heartbeat from core client for 30 sec - exiting 08:18:56 (2196): No heartbeat from core client for 30 sec - exiting 08:18:57 (2196): No heartbeat from core client for 30 sec - exiting 08:18:58 (2196): No heartbeat from core client for 30 sec - exiting 08:18:59 (2196): No heartbeat from core client for 30 sec - exiting 08:19:00 (2196): No heartbeat from core client for 30 sec - exiting 08:19:01 (2196): No heartbeat from core client for 30 sec - exiting 08:19:02 (2196): No heartbeat from core client for 30 sec - exiting 08:19:04 (2196): No heartbeat from core client for 30 sec - exiting 08:19:05 (2196): No heartbeat from core client for 30 sec - exiting 08:19:06 (2196): No heartbeat from core client for 30 sec - exiting 08:19:07 (2196): No heartbeat from core client for 30 sec - exiting 08:19:08 (2196): No heartbeat from core client for 30 sec - exiting 08:19:09 (2196): No heartbeat from core client for 30 sec - exiting 08:19:10 (2196): No heartbeat from core client for 30 sec - exiting 08:19:11 (2196): No heartbeat from core client for 30 sec - exiting 08:19:12 (2196): No heartbeat from core client for 30 sec - exiting 08:19:13 (2196): No heartbeat from core client for 30 sec - exiting 08:19:14 (2196): No heartbeat from core client for 30 sec - exiting 08:19:16 (2196): No heartbeat from core client for 30 sec - exiting 08:19:17 (2196): No heartbeat from core client for 30 sec - exiting 08:19:18 (2196): No heartbeat from core client for 30 sec - exiting 08:19:19 (2196): No heartbeat from core client for 30 sec - exiting 08:19:20 (2196): No heartbeat from core client for 30 sec - exiting 08:19:21 (2196): No heartbeat from core client for 30 sec - exiting 08:19:22 (2196): No heartbeat from core client for 30 sec - exiting 08:19:23 (2196): No heartbeat from core client for 30 sec - exiting 08:19:24 (2196): No heartbeat from core client for 30 sec - exiting 08:19:25 (2196): No heartbeat from core client for 30 sec - exiting 08:19:26 (2196): No heartbeat from core client for 30 sec - exiting 08:19:28 (2196): No heartbeat from core client for 30 sec - exiting 08:19:29 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:30 (2196): No heartbeat from core client for 30 sec - exiting 08:20:55 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:33 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Error converting file to netcdf: dataout/p78xko.pjf6c10 Error converting file to netcdf: dataout/p78xko.pif6c10 Error converting file to netcdf: dataout/p78xko.pff6c10 Error converting file to netcdf: dataout/p78xka.phf6c10 Error converting file to netcdf: dataout/p78xka.pgf6c10 Error converting file to netcdf: dataout/p78xka.pef6c10 Error converting file to netcdf: dataout/p78xka.pdf6c10 MainError: 12:44:11 AM No files match the supplied pattern. MainError: 12:44:11 AM No files match the supplied pattern. 12:02:07 (5572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Error converting file to netcdf: dataout/p78xko.pjf7c10 Error converting file to netcdf: dataout/p78xko.pif7c10 Error converting file to netcdf: dataout/p78xko.pff7c10 Error converting file to netcdf: dataout/p78xka.phf7c10 Error converting file to netcdf: dataout/p78xka.pgf7c10 Error converting file to netcdf: dataout/p78xka.pef7c10 Error converting file to netcdf: dataout/p78xka.pdf7c10 MainError: 07:17:17 AM No files match the supplied pattern. MainError: 07:17:17 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:13:07 (5280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:47 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:48 (5428): No heartbeat from core client for 30 sec - exiting 09:51:49 (5428): No heartbeat from core client for 30 sec - exiting 09:51:51 (5428): No heartbeat from core client for 30 sec - exiting 09:51:52 (5428): No heartbeat from core client for 30 sec - exiting Error converting file to netcdf: dataout/p78xko.pjf8c10 Error converting file to netcdf: dataout/p78xko.pif8c10 Error converting file to netcdf: dataout/p78xko.pff8c10 Error converting file to netcdf: dataout/p78xka.phf8c10 Error converting file to netcdf: dataout/p78xka.pgf8c10 Error converting file to netcdf: dataout/p78xka.pef8c10 Error converting file to netcdf: dataout/p78xka.pdf8c10 MainError: 02:39:57 AM No files match the supplied pattern. MainError: 02:39:57 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... 11:35:03 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:15 (2372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:56:29 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5780, iMonCtr=1 Model crash detected, will try to restart... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Nov 2011 02:42:57	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	466,560	430,587	0.9229
03 Nov 2011 07:20:51	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	440,640	410,120	0.9307
03 Nov 2011 00:47:01	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	414,720	388,755	0.9374
02 Nov 2011 07:22:52	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	388,800	367,558	0.9454
01 Nov 2011 23:13:21	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	362,880	346,191	0.9540
01 Nov 2011 00:24:21	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	336,960	324,282	0.9624
31 Oct 2011 19:41:48	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	311,040	301,658	0.9698
31 Oct 2011 19:29:25	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	285,120	278,852	0.9780
31 Oct 2011 17:40:18	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	259,200	257,677	0.9941
31 Oct 2011 16:55:40	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	233,280	236,818	1.0152
31 Oct 2011 16:24:38	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	207,360	215,846	1.0409
31 Oct 2011 15:37:41	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	181,440	195,527	1.0776
31 Oct 2011 13:38:48	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	155,520	174,972	1.1251
31 Oct 2011 13:38:48	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	129,600	154,882	1.1951
31 Oct 2011 13:38:47	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	103,680	134,411	1.2964
31 Oct 2011 13:38:47	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	77,760	114,003	1.4661
18 Oct 2011 08:51:48	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	51,840	41,621	0.8029
17 Oct 2011 04:01:47	1175495	13500098	hadcm3n_p78x_1940_40_007421653_2	25,920	20,722	0.7995