Name | hadcm3n_o4ie_2140_40_008269885_3 |
Workunit | 8425009 |
Created | 18 Jan 2013, 7:22:19 UTC |
Sent | 18 Jan 2013, 7:22:46 UTC |
Report deadline | 19 Apr 2013, 14:49:57 UTC |
Received | 17 Feb 2013, 22:59:39 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1225070 |
Run time | 11 days 16 hours 13 min 16 sec |
CPU time | 10 days 18 hours 25 min 40 sec |
Validate state | Invalid |
Credit | 9,331.20 |
Device peak FLOPS | 3.34 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86 |
Stderr | <core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5484, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3848, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5200, iMonCtr=1 Model crash detected, will try to restart... 21:02:09 (4320): No heartbeat from core client for 30 sec - exiting 21:03:37 (4320): No heartbeat from core client for 30 sec - exiting 21:03:38 (4320): No heartbeat from core client for 30 sec - exiting 21:03:39 (4320): No heartbeat from core client for 30 sec - exiting 21:03:40 (4320): No heartbeat from core client for 30 sec - exiting 21:03:41 (4320): No heartbeat from core client for 30 sec - exiting 21:03:42 (4320): No heartbeat from core client for 30 sec - exiting 21:03:43 (4320): No heartbeat from core client for 30 sec - exiting 21:03:44 (4320): No heartbeat from core client for 30 sec - exiting 21:03:45 (4320): No heartbeat from core client for 30 sec - exiting 21:03:46 (4320): No heartbeat from core client for 30 sec - exiting 21:03:47 (4320): No heartbeat from core client for 30 sec - exiting 21:03:48 (4320): No heartbeat from core client for 30 sec - exiting 21:03:49 (4320): No heartbeat from core client for 30 sec - exiting 21:03:50 (4320): No heartbeat from core client for 30 sec - exiting 21:03:51 (4320): No heartbeat from core client for 30 sec - exiting 21:03:52 (4320): No heartbeat from core client for 30 sec - exiting 21:03:53 (4320): No heartbeat from core client for 30 sec - exiting 21:03:54 (4320): No heartbeat from core client for 30 sec - exiting 21:03:55 (4320): No heartbeat from core client for 30 sec - exiting 21:03:56 (4320): No heartbeat from core client for 30 sec - exiting 21:03:57 (4320): No heartbeat from core client for 30 sec - exiting 21:03:58 (4320): No heartbeat from core client for 30 sec - exiting 21:03:59 (4320): No heartbeat from core client for 30 sec - exiting 21:04:00 (4320): No heartbeat from core client for 30 sec - exiting 21:04:01 (4320): No heartbeat from core client for 30 sec - exiting 21:04:02 (4320): No heartbeat from core client for 30 sec - exiting 21:04:03 (4320): No heartbeat from core client for 30 sec - exiting 21:04:04 (4320): No heartbeat from core client for 30 sec - exiting 21:04:05 (4320): No heartbeat from core client for 30 sec - exiting 21:04:06 (4320): No heartbeat from core client for 30 sec - exiting 21:04:07 (4320): No heartbeat from core client for 30 sec - exiting 21:04:08 (4320): No heartbeat from core client for 30 sec - exiting 21:04:09 (4320): No heartbeat from core client for 30 sec - exiting 21:04:10 (4320): No heartbeat from core client for 30 sec - exiting 21:04:11 (4320): No heartbeat from core client for 30 sec - exiting 21:04:12 (4320): No heartbeat from core client for 30 sec - exiting 21:04:13 (4320): No heartbeat from core client for 30 sec - exiting 21:04:14 (4320): No heartbeat from core client for 30 sec - exiting 21:04:15 (4320): No heartbeat from core client for 30 sec - exiting 21:04:16 (4320): No heartbeat from core client for 30 sec - exiting 21:04:17 (4320): No heartbeat from core client for 30 sec - exiting 21:04:18 (4320): No heartbeat from core client for 30 sec - exiting 21:04:19 (4320): No heartbeat from core client for 30 sec - exiting 21:04:20 (4320): No heartbeat from core client for 30 sec - exiting 21:04:21 (4320): No heartbeat from core client for 30 sec - exiting 21:04:22 (4320): No heartbeat from core client for 30 sec - exiting 21:04:23 (4320): No heartbeat from core client for 30 sec - exiting 21:04:24 (4320): No heartbeat from core client for 30 sec - exiting 21:04:25 (4320): No heartbeat from core client for 30 sec - exiting 21:04:26 (4320): No heartbeat from core client for 30 sec - exiting 21:04:27 (4320): No heartbeat from core client for 30 sec - exiting 21:04:28 (4320): No heartbeat from core client for 30 sec - exiting 21:04:29 (4320): No heartbeat from core client for 30 sec - exiting 21:04:30 (4320): No heartbeat from core client for 30 sec - exiting 21:04:31 (4320): No heartbeat from core client for 30 sec - exiting 21:04:32 (4320): No heartbeat from core client for 30 sec - exiting 21:04:33 (4320): No heartbeat from core client for 30 sec - exiting 21:04:34 (4320): No heartbeat from core client for 30 sec - exiting 21:04:35 (4320): No heartbeat from core client for 30 sec - exiting 21:04:36 (4320): No heartbeat from core client for 30 sec - exiting 21:04:37 (4320): No heartbeat from core client for 30 sec - exiting 21:04:38 (4320): No heartbeat from core client for 30 sec - exiting 21:04:39 (4320): No heartbeat from core client for 30 sec - exiting 21:04:40 (4320): No heartbeat from core client for 30 sec - exiting 21:04:41 (4320): No heartbeat from core client for 30 sec - exiting 21:04:42 (4320): No heartbeat from core client for 30 sec - exiting 21:04:43 (4320): No heartbeat from core client for 30 sec - exiting 21:04:44 (4320): No heartbeat from core client for 30 sec - exiting 21:04:45 (4320): No heartbeat from core client for 30 sec - exiting 21:04:46 (4320): No heartbeat from core client for 30 sec - exiting 21:04:47 (4320): No heartbeat from core client for 30 sec - exiting 21:04:48 (4320): No heartbeat from core client for 30 sec - exiting 21:04:49 (4320): No heartbeat from core client for 30 sec - exiting 21:04:50 (4320): No heartbeat from core client for 30 sec - exiting 21:04:51 (4320): No heartbeat from core client for 30 sec - exiting 21:04:52 (4320): No heartbeat from core client for 30 sec - exiting 21:04:53 (4320): No heartbeat from core client for 30 sec - exiting 21:04:54 (4320): No heartbeat from core client for 30 sec - exiting 21:04:55 (4320): No heartbeat from core client for 30 sec - exiting 21:04:56 (4320): No heartbeat from core client for 30 sec - exiting 21:04:57 (4320): No heartbeat from core client for 30 sec - exiting 21:04:58 (4320): No heartbeat from core client for 30 sec - exiting 21:04:59 (4320): No heartbeat from core client for 30 sec - exiting 21:05:00 (4320): No heartbeat from core client for 30 sec - exiting 21:05:01 (4320): No heartbeat from core client for 30 sec - exiting 21:05:02 (4320): No heartbeat from core client for 30 sec - exiting 21:05:03 (4320): No heartbeat from core client for 30 sec - exiting 21:05:04 (4320): No heartbeat from core client for 30 sec - exiting 21:05:05 (4320): No heartbeat from core client for 30 sec - exiting 21:05:06 (4320): No heartbeat from core client for 30 sec - exiting 21:05:07 (4320): No heartbeat from core client for 30 sec - exiting 21:05:08 (4320): No heartbeat from core client for 30 sec - exiting 21:05:09 (4320): No heartbeat from core client for 30 sec - exiting 21:05:10 (4320): No heartbeat from core client for 30 sec - exiting 21:05:11 (4320): No heartbeat from core client for 30 sec - exiting 21:05:12 (4320): No heartbeat from core client for 30 sec - exiting 21:05:13 (4320): No heartbeat from core client for 30 sec - exiting 21:05:14 (4320): No heartbeat from core client for 30 sec - exiting 21:05:15 (4320): No heartbeat from core client for 30 sec - exiting 21:05:16 (4320): No heartbeat from core client for 30 sec - exiting 21:05:17 (4320): No heartbeat from core client for 30 sec - exiting 21:05:18 (4320): No heartbeat from core client for 30 sec - exiting 21:05:19 (4320): No heartbeat from core client for 30 sec - exiting 21:05:20 (4320): No heartbeat from core client for 30 sec - exiting 21:05:21 (4320): No heartbeat from core client for 30 sec - exiting 21:05:22 (4320): No heartbeat from core client for 30 sec - exiting 21:05:23 (4320): No heartbeat from core client for 30 sec - exiting 21:05:24 (4320): No heartbeat from core client for 30 sec - exiting 21:05:25 (4320): No heartbeat from core client for 30 sec - exiting 21:05:26 (4320): No heartbeat from core client for 30 sec - exiting 21:05:27 (4320): No heartbeat from core client for 30 sec - exiting 21:05:28 (4320): No heartbeat from core client for 30 sec - exiting 21:05:29 (4320): No heartbeat from core client for 30 sec - exiting 21:05:30 (4320): No heartbeat from core client for 30 sec - exiting 21:05:31 (4320): No heartbeat from core client for 30 sec - exiting 21:05:32 (4320): No heartbeat from core client for 30 sec - exiting 21:05:33 (4320): No heartbeat from core client for 30 sec - exiting 21:05:34 (4320): No heartbeat from core client for 30 sec - exiting 21:05:35 (4320): No heartbeat from core client for 30 sec - exiting 21:05:36 (4320): No heartbeat from core client for 30 sec - exiting 21:05:37 (4320): No heartbeat from core client for 30 sec - exiting 21:05:38 (4320): No heartbeat from core client for 30 sec - exiting 21:05:39 (4320): No heartbeat from core client for 30 sec - exiting 21:05:40 (4320): No heartbeat from core client for 30 sec - exiting 21:05:41 (4320): No heartbeat from core client for 30 sec - exiting 21:05:42 (4320): No heartbeat from core client for 30 sec - exiting 21:05:43 (4320): No heartbeat from core client for 30 sec - exiting 21:05:44 (4320): No heartbeat from core client for 30 sec - exiting 21:05:45 (4320): No heartbeat from core client for 30 sec - exiting 21:05:46 (4320): No heartbeat from core client for 30 sec - exiting 21:05:47 (4320): No heartbeat from core client for 30 sec - exiting 21:05:48 (4320): No heartbeat from core client for 30 sec - exiting 21:05:49 (4320): No heartbeat from core client for 30 sec - exiting 21:05:50 (4320): No heartbeat from core client for 30 sec - exiting 21:05:51 (4320): No heartbeat from core client for 30 sec - exiting 21:05:52 (4320): No heartbeat from core client for 30 sec - exiting 21:05:53 (4320): No heartbeat from core client for 30 sec - exiting 21:05:54 (4320): No heartbeat from core client for 30 sec - exiting 21:05:55 (4320): No heartbeat from core client for 30 sec - exiting 21:05:56 (4320): No heartbeat from core client for 30 sec - exiting 21:05:57 (4320): No heartbeat from core client for 30 sec - exiting 21:05:58 (4320): No heartbeat from core client for 30 sec - exiting 21:05:59 (4320): No heartbeat from core client for 30 sec - exiting 21:06:00 (4320): No heartbeat from core client for 30 sec - exiting 21:06:01 (4320): No heartbeat from core client for 30 sec - exiting 21:06:02 (4320): No heartbeat from core client for 30 sec - exiting 21:06:03 (4320): No heartbeat from core client for 30 sec - exiting 21:06:04 (4320): No heartbeat from core client for 30 sec - exiting 21:06:05 (4320): No heartbeat from core client for 30 sec - exiting 21:06:06 (4320): No heartbeat from core client for 30 sec - exiting 21:06:07 (4320): No heartbeat from core client for 30 sec - exiting 21:06:08 (4320): No heartbeat from core client for 30 sec - exiting 21:06:09 (4320): No heartbeat from core client for 30 sec - exiting 21:06:10 (4320): No heartbeat from core client for 30 sec - exiting 21:06:11 (4320): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 05:43:49 AM No files match the supplied pattern. MainError: 05:43:49 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2228, iMonCtr=1 Model crash detected, will try to restart... MainError: 04:50:05 AM No files match the supplied pattern. MainError: 04:50:05 AM No files match the supplied pattern. 22:43:04 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:50:22 (1556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 03:35:06 AM No files match the supplied pattern. MainError: 03:35:06 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 06:35:42 PM No files match the supplied pattern. MainError: 06:35:42 PM No files match the supplied pattern. MainError: 01:22:12 AM No files match the supplied pattern. MainError: 01:22:12 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 09:32:50 AM No files match the supplied pattern. MainError: 09:32:50 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3756, iMonCtr=1 Model crash detected, will try to restart... MainError: 04:35:26 AM No files match the supplied pattern. MainError: 04:35:26 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:16:06 AM No files match the supplied pattern. MainError: 08:16:06 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 06:51:32 AM No files match the supplied pattern. MainError: 06:51:32 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4404, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... MainError: 04:51:22 AM No files match the supplied pattern. MainError: 04:51:22 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/o4ieka.ph11c10 Error converting file to netcdf: dataout/o4ieka.pg11c10 Error converting file to netcdf: dataout/o4ieka.pe11c10 MainError: 07:40:39 PM No files match the supplied pattern. MainError: 07:40:39 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
17 Feb 2013 19:43:10 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 777,600 | 935,004 | 1.2024 |
17 Feb 2013 04:56:12 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 751,680 | 902,170 | 1.2002 |
15 Feb 2013 06:51:46 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 725,760 | 869,615 | 1.1982 |
14 Feb 2013 03:34:37 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 699,840 | 838,150 | 1.1976 |
12 Feb 2013 05:28:55 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 673,920 | 807,684 | 1.1985 |
11 Feb 2013 09:56:26 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 648,000 | 778,900 | 1.2020 |
11 Feb 2013 01:49:40 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 622,080 | 750,533 | 1.2065 |
10 Feb 2013 18:41:37 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 596,160 | 726,170 | 1.2181 |
10 Feb 2013 04:20:55 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 570,240 | 693,717 | 1.2165 |
08 Feb 2013 05:05:43 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 544,320 | 660,308 | 1.2131 |
06 Feb 2013 06:27:43 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 518,400 | 626,764 | 1.2090 |
05 Feb 2013 02:35:37 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 492,480 | 593,299 | 1.2047 |
04 Feb 2013 11:33:22 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 466,560 | 559,777 | 1.1998 |
04 Feb 2013 00:10:47 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 440,640 | 526,609 | 1.1951 |
03 Feb 2013 07:07:17 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 414,720 | 493,139 | 1.1891 |
01 Feb 2013 07:26:34 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 388,800 | 459,624 | 1.1822 |
31 Jan 2013 05:20:04 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 362,880 | 432,332 | 1.1914 |
30 Jan 2013 04:52:39 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 336,960 | 407,007 | 1.2079 |
29 Jan 2013 04:13:49 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 311,040 | 376,860 | 1.2116 |
28 Jan 2013 08:54:33 | 1225070 | 15550758 | hadcm3n_o4ie_2140_40_008269885_3 | 285,120 | 345,717 | 1.2125 |
©2024 cpdn.org