climateprediction.net home page
Task 15893837

Task 15893837

Name hadcm3n_z8e3_1960_40_008350551_1
Workunit 8501412
Created 18 Jul 2013, 10:52:58 UTC
Sent 18 Jul 2013, 11:29:23 UTC
Report deadline 17 Oct 2013, 18:56:34 UTC
Received 14 Aug 2013, 16:37:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1101528
Run time 18 days 9 hours 47 min 26 sec
CPU time 16 days 23 hours 7 min 51 sec
Validate state Invalid
Credit 10,575.36
Device peak FLOPS 3.14 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
17:36:12 (51253): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:14:20 (91909): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
20:30:12 (73756): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
19:11:20 (70775): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Mon Aug  5 19:11:42 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84968, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Mon Aug  5 19:11:46 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84968, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Mon Aug  5 19:11:49 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84968, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Mon Aug  5 19:11:52 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=84968, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Mon Aug  5 19:11:55 2013

Thread 0 Crashed:
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Aug  7 10:54:58 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=294, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Aug  7 10:55:01 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x93ffcc5a ___kill + 10
1   libSystem.B.dylib                   0x9408f5a5 _raise + 26
2   libSystem.B.dylib                   0x940a56e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x93f9bc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x94023429 _nanosleep + 345
2   libSystem.B.dylib                   0x940232ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x93fc9259 __pthread_start + 345
6   libSystem.B.dylib                   0x93fc90de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc928 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x93f9b000 - 0x94142fff /usr/lib/libSystem.B.dylib
0x94c6f000 - 0x94c7dfff /usr/lib/libz.1.dylib
0x95602000 - 0x95605fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=294, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 881,280 1,455,942 1.6521
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 855,360 1,413,680 1.6527
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 829,440 1,371,454 1.6535
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 803,520 1,329,384 1.6545
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 777,600 1,286,878 1.6549
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 751,680 1,245,522 1.6570
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 725,760 1,203,226 1.6579
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 699,840 1,161,040 1.6590
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 673,920 1,119,115 1.6606
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 648,000 1,076,689 1.6616
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 622,080 1,034,236 1.6625
14 Aug 2013 16:40:28 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 596,160 992,478 1.6648
30 Jul 2013 09:52:21 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 570,240 950,028 1.6660
30 Jul 2013 09:52:21 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 544,320 907,427 1.6671
29 Jul 2013 13:01:27 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 518,400 864,890 1.6684
29 Jul 2013 13:01:27 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 492,480 821,219 1.6675
29 Jul 2013 13:01:27 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 466,560 778,260 1.6681
29 Jul 2013 13:01:27 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 440,640 734,996 1.6680
29 Jul 2013 13:01:27 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 414,720 691,905 1.6684
26 Jul 2013 10:51:01 1101528 15893837 hadcm3n_z8e3_1960_40_008350551_1 388,800 648,450 1.6678


©2024 climateprediction.net