Name | hadcm3n_871n_1980_40_008515634_1 |
Workunit | 8663146 |
Created | 29 May 2014, 0:12:57 UTC |
Sent | 29 May 2014, 0:13:37 UTC |
Report deadline | 28 Aug 2014, 7:40:48 UTC |
Received | 1 Jul 2014, 12:38:25 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 22 (0x00000016) Unknown error code |
Computer ID | 1322748 |
Run time | 16 days 2 hours 48 min 11 sec |
CPU time | 15 days 11 hours 28 min 34 sec |
Validate state | Invalid |
Credit | 10,886.40 |
Device peak FLOPS | 2.83 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86 |
Stderr | <core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9728, selfPID=9728, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5700, selfPID=5700, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16100, selfPID=16100, iMonCtr=1 forrtl: The requested operation cannot be performed on a file with a user-mapped section open. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13880, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 07:49:04 (12264): No heartbeat from core client for 30 sec - exiting 07:49:05 (12264): No heartbeat from core client for 30 sec - exiting 07:49:06 (12264): No heartbeat from core client for 30 sec - exiting 07:49:07 (12264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=612, selfPID=612, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=18904, selfPID=18904, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14380, selfPID=14380, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8008, selfPID=8008, iMonCtr=1 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15404, selfPID=15404, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8828, selfPID=8828, iMonCtr=1 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9968, selfPID=9968, iMonCtr=1 22:39:24 (10912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15444, selfPID=15444, iMonCtr=1 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=21448, selfPID=21448, iMonCtr=1 Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:28:16 (8084): No heartbeat from core client for 30 sec - exiting 15:28:17 (8084): No heartbeat from core client for 30 sec - exiting 15:28:18 (8084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied 11:14:07 (3376): No heartbeat from core client for 30 sec - exiting 11:14:13 (3376): No heartbeat from core client for 30 sec - exiting 11:14:14 (3376): No heartbeat from core client for 30 sec - exiting 11:14:54 (3376): No heartbeat from core client for 30 sec - exiting 11:14:55 (3376): No heartbeat from core client for 30 sec - exiting 11:14:56 (3376): No heartbeat from core client for 30 sec - exiting 11:14:57 (3376): No heartbeat from core client for 30 sec - exiting 11:14:58 (3376): No heartbeat from core client for 30 sec - exiting 11:14:59 (3376): No heartbeat from core client for 30 sec - exiting 11:15:00 (3376): No heartbeat from core client for 30 sec - exiting 11:15:01 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied 12:06:06 (1156): No heartbeat from core client for 30 sec - exiting 12:06:07 (1156): No heartbeat from core client for 30 sec - exiting 12:06:08 (1156): No heartbeat from core client for 30 sec - exiting 12:06:09 (1156): No heartbeat from core client for 30 sec - exiting 12:06:10 (1156): No heartbeat from core client for 30 sec - exiting 12:06:11 (1156): No heartbeat from core client for 30 sec - exiting 12:06:12 (1156): No heartbeat from core client for 30 sec - exiting 12:06:13 (1156): No heartbeat from core client for 30 sec - exiting 12:06:14 (1156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Skipping gmts_generator due to netcdf error 13 - Permission denied Suspended CPDN Monitor - Suspend request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Skipping gmts_generator due to netcdf error 13 - Permission denied CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:49:57 (4216): No heartbeat from core client for 30 sec - exiting 08:49:58 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13300, selfPID=13300, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... 11:09:13 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14132, selfPID=14132, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12348, selfPID=12348, iMonCtr=1 13:21:52 (9516): No heartbeat from core client for 30 sec - exiting 13:21:53 (9516): No heartbeat from core client for 30 sec - exiting 13:21:54 (9516): No heartbeat from core client for 30 sec - exiting 13:21:55 (9516): No heartbeat from core client for 30 sec - exiting 13:21:56 (9516): No heartbeat from core client for 30 sec - exiting 13:21:58 (9516): No heartbeat from core client for 30 sec - exiting 13:21:59 (9516): No heartbeat from core client for 30 sec - exiting 13:22:00 (9516): No heartbeat from core client for 30 sec - exiting 13:22:01 (9516): No heartbeat from core client for 30 sec - exiting 13:22:02 (9516): No heartbeat from core client for 30 sec - exiting 13:22:03 (9516): No heartbeat from core client for 30 sec - exiting 13:22:04 (9516): No heartbeat from core client for 30 sec - exiting 13:22:05 (9516): No heartbeat from core client for 30 sec - exiting 13:22:06 (9516): No heartbeat from core client for 30 sec - exiting 13:22:07 (9516): No heartbeat from core client for 30 sec - exiting 13:22:09 (9516): No heartbeat from core client for 30 sec - exiting 13:22:10 (9516): No heartbeat from core client for 30 sec - exiting 13:22:11 (9516): No heartbeat from core client for 30 sec - exiting 13:22:12 (9516): No heartbeat from core client for 30 sec - exiting 13:22:13 (9516): No heartbeat from core client for 30 sec - exiting 13:22:14 (9516): No heartbeat from core client for 30 sec - exiting 13:22:15 (9516): No heartbeat from core client for 30 sec - exiting 13:22:16 (9516): No heartbeat from core client for 30 sec - exiting 13:22:17 (9516): No heartbeat from core client for 30 sec - exiting 13:22:18 (9516): No heartbeat from core client for 30 sec - exiting 13:22:19 (9516): No heartbeat from core client for 30 sec - exiting 13:22:21 (9516): No heartbeat from core client for 30 sec - exiting 13:22:22 (9516): No heartbeat from core client for 30 sec - exiting 13:22:23 (9516): No heartbeat from core client for 30 sec - exiting 13:22:24 (9516): No heartbeat from core client for 30 sec - exiting 13:22:25 (9516): No heartbeat from core client for 30 sec - exiting 13:22:26 (9516): No heartbeat from core client for 30 sec - exiting 13:22:27 (9516): No heartbeat from core client for 30 sec - exiting 13:22:28 (9516): No heartbeat from core client for 30 sec - exiting 13:22:29 (9516): No heartbeat from core client for 30 sec - exiting 13:22:30 (9516): No heartbeat from core client for 30 sec - exiting 13:22:31 (9516): No heartbeat from core client for 30 sec - exiting 13:22:33 (9516): No heartbeat from core client for 30 sec - exiting 13:22:34 (9516): No heartbeat from core client for 30 sec - exiting 13:22:35 (9516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4836, selfPID=4836, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7132, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
01 Jul 2014 07:51:07 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 907,200 | 1,305,066 | 1.4386 |
01 Jul 2014 07:51:07 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 881,280 | 1,270,865 | 1.4421 |
28 Jun 2014 10:50:38 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 855,360 | 1,235,692 | 1.4446 |
28 Jun 2014 00:47:39 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 829,440 | 1,199,536 | 1.4462 |
27 Jun 2014 04:30:44 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 803,520 | 1,168,139 | 1.4538 |
26 Jun 2014 05:27:30 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 777,600 | 1,124,886 | 1.4466 |
25 Jun 2014 02:41:38 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 751,680 | 1,077,634 | 1.4336 |
24 Jun 2014 04:31:42 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 725,760 | 1,038,598 | 1.4310 |
21 Jun 2014 12:51:11 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 699,840 | 995,801 | 1.4229 |
20 Jun 2014 22:53:59 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 673,920 | 952,052 | 1.4127 |
19 Jun 2014 10:41:51 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 648,000 | 907,731 | 1.4008 |
18 Jun 2014 07:09:02 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 622,080 | 863,990 | 1.3889 |
17 Jun 2014 17:47:15 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 596,160 | 821,591 | 1.3781 |
17 Jun 2014 09:56:10 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 570,240 | 793,633 | 1.3918 |
17 Jun 2014 04:35:21 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 544,320 | 772,217 | 1.4187 |
16 Jun 2014 22:14:41 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 518,400 | 750,913 | 1.4485 |
16 Jun 2014 15:49:00 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 492,480 | 727,320 | 1.4769 |
16 Jun 2014 04:12:57 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 466,560 | 703,318 | 1.5075 |
15 Jun 2014 16:16:53 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 440,640 | 679,401 | 1.5419 |
15 Jun 2014 08:36:15 | 1322748 | 16649891 | hadcm3n_871n_1980_40_008515634_1 | 414,720 | 652,084 | 1.5723 |
©2024 cpdn.org