climateprediction.net home page
Task 15918562

Task 15918562

Name hadcm3n_3dq0_1980_40_008366680_2
Workunit 8517539
Created 14 Aug 2013, 16:19:09 UTC
Sent 14 Aug 2013, 19:12:27 UTC
Report deadline 14 Nov 2013, 2:39:38 UTC
Received 13 Sep 2013, 1:02:20 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1231328
Run time 16 days 6 hours 25 min 44 sec
CPU time 9 days 18 hours 34 min 30 sec
Validate state Invalid
Credit 5,287.68
Device peak FLOPS 3.01 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
03:36:30 (70541): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:36:31 (70541): No heartbeat from core client for 30 sec - exiting
03:36:32 (70541): No heartbeat from core client for 30 sec - exiting
03:36:33 (70541): No heartbeat from core client for 30 sec - exiting
03:36:34 (70541): No heartbeat from core client for 30 sec - exiting
03:36:35 (70541): No heartbeat from core client for 30 sec - exiting
03:36:36 (70541): No heartbeat from core client for 30 sec - exiting
03:36:37 (70541): No heartbeat from core client for 30 sec - exiting
03:36:38 (70541): No heartbeat from core client for 30 sec - exiting
03:36:39 (70541): No heartbeat from core client for 30 sec - exiting
03:36:40 (70541): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
07:50:06 (489): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:50:07 (489): No heartbeat from core client for 30 sec - exiting
07:50:08 (489): No heartbeat from core client for 30 sec - exiting
07:50:09 (489): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
03:43:55 (20440): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:43:57 (20440): No heartbeat from core client for 30 sec - exiting
03:43:58 (20440): No heartbeat from core client for 30 sec - exiting
03:43:59 (20440): No heartbeat from core client for 30 sec - exiting
03:44:00 (20440): No heartbeat from core client for 30 sec - exiting
03:44:01 (20440): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
23:17:23 (41129): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:18:25 (43284): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:18:27 (43284): No heartbeat from core client for 30 sec - exiting
03:19:12 (44503): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
07:53:21 (61063): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:53:23 (61063): No heartbeat from core client for 30 sec - exiting
07:53:24 (61063): No heartbeat from core client for 30 sec - exiting
07:53:25 (61063): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
00:50:24 (70958): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
20:49:58 (1645): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
01:24:05 (2865): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:19:30 (4700): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
21:36:21 (11634): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:38:24 (13115): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:38:26 (13115): No heartbeat from core client for 30 sec - exiting
05:38:27 (13115): No heartbeat from core client for 30 sec - exiting
05:38:28 (13115): No heartbeat from core client for 30 sec - exiting
05:38:29 (13115): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
02:05:31 (31470): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:05:33 (31470): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
23:48:28 (71677): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:10:07 (74032): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
21:36:02 (81748): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:56:19 (83255): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:56:21 (83255): No heartbeat from core client for 30 sec - exiting
02:56:22 (83255): No heartbeat from core client for 30 sec - exiting
02:56:23 (83255): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
19:28:55 (91578): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:28:58 (91578): No heartbeat from core client for 30 sec - exiting
19:28:59 (91578): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
22:37:04 (1554): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:27:11 (3438): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:42:18 (3786): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:42:20 (3786): No heartbeat from core client for 30 sec - exiting
00:42:21 (3786): No heartbeat from core client for 30 sec - exiting
00:42:22 (3786): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
00:20:53 (31169): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
21:33:18 (41921): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:31 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:35 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:38 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:41 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:44 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=43370, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Sep 11 21:33:46 2013

Thread 0 Crashed:
CPDN Monitor - Quit request from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Thu Sep 12 18:00:02 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x95089c5a __kill + 10
1   libSystem.B.dylib                   0x9511c5a5 raise + 26
2   libSystem.B.dylib                   0x951326e4 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x95028c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x950b0429 nanosleep + 345
2   libSystem.B.dylib                   0x950b02ca usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x95056259 _pthread_start + 345
6   libSystem.B.dylib                   0x950560de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc958 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x9035a000 - 0x90368fff /usr/lib/libz.1.dylib
0x90d16000 - 0x90d19fff /usr/lib/system/libmathCommon.A.dylib
0x95028000 - 0x951cffff /usr/lib/libSystem.B.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=51866, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Sep 2013 01:34:12 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 440,640 808,642 1.8352
09 Sep 2013 06:52:37 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 414,720 760,641 1.8341
07 Sep 2013 12:04:54 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 388,800 712,245 1.8319
06 Sep 2013 03:25:06 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 362,880 664,031 1.8299
04 Sep 2013 08:56:46 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 336,960 616,020 1.8282
02 Sep 2013 14:21:35 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 311,040 567,862 1.8257
01 Sep 2013 05:23:00 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 285,120 519,324 1.8214
30 Aug 2013 12:42:04 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 259,200 473,653 1.8274
29 Aug 2013 04:53:14 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 233,280 425,498 1.8240
27 Aug 2013 08:35:39 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 207,360 376,594 1.8161
25 Aug 2013 14:59:22 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 181,440 328,805 1.8122
24 Aug 2013 06:02:29 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 155,520 280,749 1.8052
22 Aug 2013 15:26:36 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 129,600 233,840 1.8043
21 Aug 2013 06:31:14 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 103,680 189,983 1.8324
19 Aug 2013 14:40:53 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 77,760 145,870 1.8759
18 Aug 2013 05:24:00 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 51,840 97,175 1.8745
16 Aug 2013 10:39:57 1231328 15918562 hadcm3n_3dq0_1980_40_008366680_2 25,920 48,650 1.8769


©2024 cpdn.org