Name | hadcm3n_ybf8_1940_40_007538488_1 |
Workunit | 7735720 |
Created | 5 Nov 2011, 21:15:13 UTC |
Sent | 9 Nov 2011, 20:54:49 UTC |
Report deadline | 9 Feb 2012, 4:22:00 UTC |
Received | 1 Jan 2012, 18:16:26 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 193 (0x000000C1) EXIT_SIGNAL |
Computer ID | 1021113 |
Run time | 18 days 22 hours 15 min 1 sec |
CPU time | 18 days 0 hours 30 min 34 sec |
Validate state | Invalid |
Credit | 9,331.20 |
Device peak FLOPS | 2.52 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86 |
Stderr | <core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7476, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4500, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3488, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1184, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5380, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4612, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5604, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:09:43 (4368): No heartbeat from core client for 30 sec - exiting 09:09:44 (4368): No heartbeat from core client for 30 sec - exiting 09:09:45 (4368): No heartbeat from core client for 30 sec - exiting 09:09:46 (4368): No heartbeat from core client for 30 sec - exiting 09:09:47 (4368): No heartbeat from core client for 30 sec - exiting 09:09:48 (4368): No heartbeat from core client for 30 sec - exiting 09:09:49 (4368): No heartbeat from core client for 30 sec - exiting 09:09:50 (4368): No heartbeat from core client for 30 sec - exiting 09:09:51 (4368): No heartbeat from core client for 30 sec - exiting 09:09:52 (4368): No heartbeat from core client for 30 sec - exiting 09:09:53 (4368): No heartbeat from core client for 30 sec - exiting 09:09:54 (4368): No heartbeat from core client for 30 sec - exiting 09:09:55 (4368): No heartbeat from core client for 30 sec - exiting 09:09:56 (4368): No heartbeat from core client for 30 sec - exiting 09:09:57 (4368): No heartbeat from core client for 30 sec - exiting 09:09:58 (4368): No heartbeat from core client for 30 sec - exiting 09:09:59 (4368): No heartbeat from core client for 30 sec - exiting 09:10:00 (4368): No heartbeat from core client for 30 sec - exiting 09:10:01 (4368): No heartbeat from core client for 30 sec - exiting 09:10:02 (4368): No heartbeat from core client for 30 sec - exiting 09:10:03 (4368): No heartbeat from core client for 30 sec - exiting 09:10:04 (4368): No heartbeat from core client for 30 sec - exiting 09:10:05 (4368): No heartbeat from core client for 30 sec - exiting 09:10:06 (4368): No heartbeat from core client for 30 sec - exiting 09:10:07 (4368): No heartbeat from core client for 30 sec - exiting 09:10:08 (4368): No heartbeat from core client for 30 sec - exiting 09:10:09 (4368): No heartbeat from core client for 30 sec - exiting 09:10:10 (4368): No heartbeat from core client for 30 sec - exiting 09:10:11 (4368): No heartbeat from core client for 30 sec - exiting 09:10:12 (4368): No heartbeat from core client for 30 sec - exiting 09:10:13 (4368): No heartbeat from core client for 30 sec - exiting 09:10:14 (4368): No heartbeat from core client for 30 sec - exiting 09:10:15 (4368): No heartbeat from core client for 30 sec - exiting 09:10:16 (4368): No heartbeat from core client for 30 sec - exiting 09:10:17 (4368): No heartbeat from core client for 30 sec - exiting 09:10:18 (4368): No heartbeat from core client for 30 sec - exiting 09:10:19 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:10:20 (4368): No heartbeat from core client for 30 sec - exiting 09:10:21 (4368): No heartbeat from core client for 30 sec - exiting 09:10:22 (4368): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5392, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:13:05 (5772): No heartbeat from core client for 30 sec - exiting 10:13:06 (5772): No heartbeat from core client for 30 sec - exiting 10:13:07 (5772): No heartbeat from core client for 30 sec - exiting 10:13:08 (5772): No heartbeat from core client for 30 sec - exiting 10:13:09 (5772): No heartbeat from core client for 30 sec - exiting 10:13:10 (5772): No heartbeat from core client for 30 sec - exiting 10:13:11 (5772): No heartbeat from core client for 30 sec - exiting 10:13:12 (5772): No heartbeat from core client for 30 sec - exiting 10:13:13 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1608, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5300, iMonCtr=1 Model crash detected, will try to restart... 16:16:56 (5788): No heartbeat from core client for 30 sec - exiting 16:16:57 (5788): No heartbeat from core client for 30 sec - exiting 16:16:58 (5788): No heartbeat from core client for 30 sec - exiting 16:16:59 (5788): No heartbeat from core client for 30 sec - exiting 16:17:00 (5788): No heartbeat from core client for 30 sec - exiting 16:17:01 (5788): No heartbeat from core client for 30 sec - exiting 16:17:02 (5788): No heartbeat from core client for 30 sec - exiting 16:17:03 (5788): No heartbeat from core client for 30 sec - exiting 16:17:04 (5788): No heartbeat from core client for 30 sec - exiting 16:17:05 (5788): No heartbeat from core client for 30 sec - exiting 16:17:06 (5788): No heartbeat from core client for 30 sec - exiting 16:17:07 (5788): No heartbeat from core client for 30 sec - exiting 16:17:08 (5788): No heartbeat from core client for 30 sec - exiting 16:17:09 (5788): No heartbeat from core client for 30 sec - exiting 16:17:10 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6644, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5856, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5268, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=1 Model crash detected, will try to restart... 12:33:44 (5296): No heartbeat from core client for 30 sec - exiting 12:33:45 (5296): No heartbeat from core client for 30 sec - exiting 12:33:46 (5296): No heartbeat from core client for 30 sec - exiting 12:33:47 (5296): No heartbeat from core client for 30 sec - exiting 12:33:48 (5296): No heartbeat from core client for 30 sec - exiting 12:33:49 (5296): No heartbeat from core client for 30 sec - exiting 12:33:50 (5296): No heartbeat from core client for 30 sec - exiting 12:33:51 (5296): No heartbeat from core client for 30 sec - exiting 12:33:52 (5296): No heartbeat from core client for 30 sec - exiting 12:33:53 (5296): No heartbeat from core client for 30 sec - exiting 12:33:54 (5296): No heartbeat from core client for 30 sec - exiting 12:33:55 (5296): No heartbeat from core client for 30 sec - exiting 12:33:56 (5296): No heartbeat from core client for 30 sec - exiting 12:33:57 (5296): No heartbeat from core client for 30 sec - exiting 12:33:58 (5296): No heartbeat from core client for 30 sec - exiting 12:33:59 (5296): No heartbeat from core client for 30 sec - exiting 12:34:00 (5296): No heartbeat from core client for 30 sec - exiting 12:34:01 (5296): No heartbeat from core client for 30 sec - exiting 12:34:02 (5296): No heartbeat from core client for 30 sec - exiting 12:34:03 (5296): No heartbeat from core client for 30 sec - exiting 12:34:04 (5296): No heartbeat from core client for 30 sec - exiting 12:34:05 (5296): No heartbeat from core client for 30 sec - exiting 12:34:06 (5296): No heartbeat from core client for 30 sec - exiting 12:34:07 (5296): No heartbeat from core client for 30 sec - exiting 12:34:08 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5612, iMonCtr=1 Model crash detected, will try to restart... 10:40:33 (5660): No heartbeat from core client for 30 sec - exiting 10:40:35 (5660): No heartbeat from core client for 30 sec - exiting 10:40:36 (5660): No heartbeat from core client for 30 sec - exiting 10:40:37 (5660): No heartbeat from core client for 30 sec - exiting 10:40:38 (5660): No heartbeat from core client for 30 sec - exiting 10:40:39 (5660): No heartbeat from core client for 30 sec - exiting 10:40:40 (5660): No heartbeat from core client for 30 sec - exiting 10:40:41 (5660): No heartbeat from core client for 30 sec - exiting 10:40:42 (5660): No heartbeat from core client for 30 sec - exiting 10:40:43 (5660): No heartbeat from core client for 30 sec - exiting 10:40:44 (5660): No heartbeat from core client for 30 sec - exiting 10:40:45 (5660): No heartbeat from core client for 30 sec - exiting 10:40:46 (5660): No heartbeat from core client for 30 sec - exiting 10:40:47 (5660): No heartbeat from core client for 30 sec - exiting 10:40:48 (5660): No heartbeat from core client for 30 sec - exiting 10:40:49 (5660): No heartbeat from core client for 30 sec - exiting 10:40:50 (5660): No heartbeat from core client for 30 sec - exiting 10:40:51 (5660): No heartbeat from core client for 30 sec - exiting 10:40:52 (5660): No heartbeat from core client for 30 sec - exiting 10:40:53 (5660): No heartbeat from core client for 30 sec - exiting 10:40:54 (5660): No heartbeat from core client for 30 sec - exiting 10:40:55 (5660): No heartbeat from core client for 30 sec - exiting 10:40:56 (5660): No heartbeat from core client for 30 sec - exiting 10:40:57 (5660): No heartbeat from core client for 30 sec - exiting 10:40:58 (5660): No heartbeat from core client for 30 sec - exiting 10:40:59 (5660): No heartbeat from core client for 30 sec - exiting 10:41:00 (5660): No heartbeat from core client for 30 sec - exiting 10:41:01 (5660): No heartbeat from core client for 30 sec - exiting 10:41:02 (5660): No heartbeat from core client for 30 sec - exiting 10:41:03 (5660): No heartbeat from core client for 30 sec - exiting 10:41:04 (5660): No heartbeat from core client for 30 sec - exiting 10:41:05 (5660): No heartbeat from core client for 30 sec - exiting 10:41:06 (5660): No heartbeat from core client for 30 sec - exiting 10:41:07 (5660): No heartbeat from core client for 30 sec - exiting 10:41:08 (5660): No heartbeat from core client for 30 sec - exiting 10:41:09 (5660): No heartbeat from core client for 30 sec - exiting 10:41:10 (5660): No heartbeat from core client for 30 sec - exiting 10:41:11 (5660): No heartbeat from core client for 30 sec - exiting 10:41:12 (5660): No heartbeat from core client for 30 sec - exiting 10:41:13 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7076, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3988, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1112, iMonCtr=1 Model crash detected, will try to restart... 11:33:05 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CB3A93 read attempt to address 0x40CB1C2B Engaging BOINC Windows Runtime Debugger... Cannot serialize file D:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_ybf8_1940_40_007538488/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
01 Jan 2012 18:43:27 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 777,600 | 1,557,029 | 2.0024 |
31 Dec 2011 16:28:09 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 751,680 | 1,505,575 | 2.0029 |
29 Dec 2011 19:29:09 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 725,760 | 1,453,166 | 2.0023 |
27 Dec 2011 19:04:46 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 699,840 | 1,399,757 | 2.0001 |
26 Dec 2011 20:13:51 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 673,920 | 1,346,904 | 1.9986 |
25 Dec 2011 18:04:21 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 648,000 | 1,294,478 | 1.9977 |
24 Dec 2011 21:55:59 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 622,080 | 1,244,057 | 1.9998 |
23 Dec 2011 18:33:13 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 596,160 | 1,192,210 | 1.9998 |
22 Dec 2011 19:02:58 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 570,240 | 1,138,901 | 1.9972 |
21 Dec 2011 12:23:48 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 544,320 | 1,087,053 | 1.9971 |
19 Dec 2011 15:19:15 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 518,400 | 1,034,666 | 1.9959 |
16 Dec 2011 16:53:54 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 492,480 | 982,524 | 1.9951 |
13 Dec 2011 15:33:15 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 466,560 | 929,418 | 1.9921 |
11 Dec 2011 10:57:18 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 440,640 | 876,673 | 1.9895 |
10 Dec 2011 09:26:13 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 414,720 | 824,999 | 1.9893 |
08 Dec 2011 15:04:44 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 388,800 | 773,393 | 1.9892 |
06 Dec 2011 13:51:26 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 362,880 | 721,766 | 1.9890 |
05 Dec 2011 14:37:45 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 336,960 | 669,036 | 1.9855 |
04 Dec 2011 11:15:32 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 311,040 | 615,368 | 1.9784 |
02 Dec 2011 20:32:14 | 1021113 | 13606566 | hadcm3n_ybf8_1940_40_007538488_1 | 285,120 | 563,265 | 1.9755 |
©2024 cpdn.org