climateprediction.net home page
Task 15813599

Task 15813599

Name hadcm3n_o3wm_1940_40_008382147_0
Workunit 8533006
Created 1 Jun 2013, 3:49:15 UTC
Sent 1 Jun 2013, 4:07:43 UTC
Report deadline 31 Aug 2013, 11:34:54 UTC
Received 21 Jul 2013, 3:53:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 650587
Run time 13 days 13 hours 29 min 25 sec
CPU time 11 days 11 hours 59 min 43 sec
Validate state Invalid
Credit 3,732.48
Device peak FLOPS 1.70 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1041200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(43580,0xa0941540) malloc: *** error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Jul 17 05:56:55 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9788ec5a ___kill + 10
1   libSystem.B.dylib                   0x979215a5 _raise + 26
2   libSystem.B.dylib                   0x979376e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9782dc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x978b5429 _nanosleep + 345
2   libSystem.B.dylib                   0x978b52ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x9785b259 __pthread_start + 345
6   libSystem.B.dylib                   0x9785b0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x902b0000 - 0x902befff /usr/lib/libz.1.dylib
0x9782d000 - 0x979d4fff /usr/lib/libSystem.B.dylib
0x99c88000 - 0x99c8bfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47388, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Jul 17 05:56:59 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9788ec5a ___kill + 10
1   libSystem.B.dylib                   0x979215a5 _raise + 26
2   libSystem.B.dylib                   0x979376e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x9782dc0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x978b5429 _nanosleep + 345
2   libSystem.B.dylib                   0x978b52ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x9785b259 __pthread_start + 345
6   libSystem.B.dylib                   0x9785b0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x902b0000 - 0x902befff /usr/lib/libz.1.dylib
0x9782d000 - 0x979d4fff /usr/lib/libSystem.B.dylib
0x99c88000 - 0x99c8bfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47388, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Wed Jul 17 05:57:06 2013

Thread 0 Crashed:
CPDN Monitor - Quit request from BOINC...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Sat Jul 20 19:36:19 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x00d22c5a ___kill + 10
1   libSystem.B.dylib                   0x00db55a5 _raise + 26
2   libSystem.B.dylib                   0x00dcb6e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x00cc1c0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x00d49429 _nanosleep + 345
2   libSystem.B.dylib                   0x00d492ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x00cef259 __pthread_start + 345
6   libSystem.B.dylib                   0x00cef0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
  0xcc1000 -   0xe68fff /usr/lib/libSystem.B.dylib
  0xeea000 -   0xef8fff /usr/lib/libz.1.dylib
  0xefd000 -   0xf00fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1938, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Sat Jul 20 19:36:22 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x00d22c5a ___kill + 10
1   libSystem.B.dylib                   0x00db55a5 _raise + 26
2   libSystem.B.dylib                   0x00dcb6e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x00cc1c0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x00d49429 _nanosleep + 345
2   libSystem.B.dylib                   0x00d492ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x00cef259 __pthread_start + 345
6   libSystem.B.dylib                   0x00cef0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
  0xcc1000 -   0xe68fff /usr/lib/libSystem.B.dylib
  0xeea000 -   0xef8fff /usr/lib/libz.1.dylib
  0xefd000 -   0xf00fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1938, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Sat Jul 20 19:36:25 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x00d22c5a ___kill + 10
1   libSystem.B.dylib                   0x00db55a5 _raise + 26
2   libSystem.B.dylib                   0x00dcb6e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x00cc1c0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x00d49429 _nanosleep + 345
2   libSystem.B.dylib                   0x00d492ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x00cef259 __pthread_start + 345
6   libSystem.B.dylib                   0x00cef0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
  0xcc1000 -   0xe68fff /usr/lib/libSystem.B.dylib
  0xeea000 -   0xef8fff /usr/lib/libz.1.dylib
  0xefd000 -   0xf00fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1938, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Sat Jul 20 19:36:28 2013

Thread 0 Crashed:
0   libSystem.B.dylib                   0x00d22c5a ___kill + 10
1   libSystem.B.dylib                   0x00db55a5 _raise + 26
2   libSystem.B.dylib                   0x00dcb6e4 _abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff _um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 _um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 _main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb __start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x00cc1c0e _mach_wait_until + 10
1   libSystem.B.dylib                   0x00d49429 _nanosleep + 345
2   libSystem.B.dylib                   0x00d492ca _usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 __Z11boinc_sleepd + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e __Z12timer_threadPv + 78
5   libSystem.B.dylib                   0x00cef259 __pthread_start + 345
6   libSystem.B.dylib                   0x00cef0de _thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc898 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
  0xcc1000 -   0xe68fff /usr/lib/libSystem.B.dylib
  0xeea000 -   0xef8fff /usr/lib/libz.1.dylib
  0xefd000 -   0xf00fff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1938, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Jul 2013 15:15:42 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 311,040 928,127 2.9839
11 Jul 2013 11:01:45 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 285,120 851,540 2.9866
08 Jul 2013 07:57:08 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 259,200 775,733 2.9928
06 Jul 2013 09:43:35 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 233,280 700,159 3.0014
02 Jul 2013 11:58:58 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 207,360 623,005 3.0045
27 Jun 2013 22:52:56 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 181,440 542,915 2.9923
26 Jun 2013 21:17:55 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 155,520 463,058 2.9775
25 Jun 2013 20:55:56 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 129,600 383,166 2.9565
13 Jun 2013 06:58:08 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 103,680 305,008 2.9418
07 Jun 2013 11:31:39 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 77,760 228,528 2.9389
05 Jun 2013 07:44:53 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 51,840 153,382 2.9588
03 Jun 2013 01:07:31 650587 15813599 hadcm3n_o3wm_1940_40_008382147_0 25,920 77,622 2.9947


©2024 climateprediction.net