climateprediction.net home page
Task 15779612

Task 15779612

Name hadcm3n_o2af_2140_40_008269113_2
Workunit 8424237
Created 12 May 2013, 23:09:58 UTC
Sent 12 May 2013, 23:10:10 UTC
Report deadline 12 Aug 2013, 6:37:21 UTC
Received 31 May 2013, 21:27:00 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1282401
Run time 18 days 11 hours 4 min 23 sec
CPU time 17 days 18 hours 43 min 15 sec
Validate state Invalid
Credit 6,842.88
Device peak FLOPS 2.01 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.27</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
15:42:45 (20848): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
17:25:14 (2317): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:25:15 (2317): No heartbeat from core client for 30 sec - exiting
17:25:16 (2317): No heartbeat from core client for 30 sec - exiting
09:35:20 (2913): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
02:02:27 (13074): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:54:28 (9882): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
MainError:	06:06:12 PM	No files match the supplied pattern.
MainError:	06:06:12 PM	No files match the supplied pattern.
06:40:34 (10434): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:40:35 (10434): No heartbeat from core client for 30 sec - exiting
06:40:36 (10434): No heartbeat from core client for 30 sec - exiting
06:40:37 (10434): No heartbeat from core client for 30 sec - exiting
06:40:38 (10434): No heartbeat from core client for 30 sec - exiting
06:40:39 (10434): No heartbeat from core client for 30 sec - exiting
06:40:40 (10434): No heartbeat from core client for 30 sec - exiting
06:40:41 (10434): No heartbeat from core client for 30 sec - exiting
06:40:42 (10434): No heartbeat from core client for 30 sec - exiting
06:40:43 (10434): No heartbeat from core client for 30 sec - exiting
06:40:44 (10434): No heartbeat from core client for 30 sec - exiting
06:40:45 (10434): No heartbeat from core client for 30 sec - exiting
06:40:46 (10434): No heartbeat from core client for 30 sec - exiting
06:40:47 (10434): No heartbeat from core client for 30 sec - exiting
06:40:48 (10434): No heartbeat from core client for 30 sec - exiting
06:40:49 (10434): No heartbeat from core client for 30 sec - exiting
06:40:50 (10434): No heartbeat from core client for 30 sec - exiting
06:40:51 (10434): No heartbeat from core client for 30 sec - exiting
06:40:52 (10434): No heartbeat from core client for 30 sec - exiting
06:40:53 (10434): No heartbeat from core client for 30 sec - exiting
06:40:54 (10434): No heartbeat from core client for 30 sec - exiting
06:40:55 (10434): No heartbeat from core client for 30 sec - exiting
06:40:56 (10434): No heartbeat from core client for 30 sec - exiting
10:36:02 (38895): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:28 (41679): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:29 (41679): No heartbeat from core client for 30 sec - exiting
10:40:30 (41679): No heartbeat from core client for 30 sec - exiting
10:40:31 (41679): No heartbeat from core client for 30 sec - exiting
14:03:47 (41895): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:24 (43968): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:25 (43968): No heartbeat from core client for 30 sec - exiting
14:13:26 (43968): No heartbeat from core client for 30 sec - exiting
14:13:27 (43968): No heartbeat from core client for 30 sec - exiting
14:13:28 (43968): No heartbeat from core client for 30 sec - exiting
14:51:27 (44344): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:05:46 (44854): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:47:26 (45269): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:31:37 (45859): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:31:38 (45859): No heartbeat from core client for 30 sec - exiting
16:31:39 (45859): No heartbeat from core client for 30 sec - exiting
17:38:37 (46494): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:38:38 (46494): No heartbeat from core client for 30 sec - exiting
17:38:39 (46494): No heartbeat from core client for 30 sec - exiting
17:38:40 (46494): No heartbeat from core client for 30 sec - exiting
17:38:41 (46494): No heartbeat from core client for 30 sec - exiting
17:38:42 (46494): No heartbeat from core client for 30 sec - exiting
17:38:43 (46494): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
18:23:57 (47646): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:22 (48374): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:23 (48374): No heartbeat from core client for 30 sec - exiting
18:28:24 (48374): No heartbeat from core client for 30 sec - exiting
MainError:	05:35:49 PM	No files match the supplied pattern.
MainError:	05:35:49 PM	No files match the supplied pattern.
19:51:14 (48563): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:01 (49526): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:02 (49526): No heartbeat from core client for 30 sec - exiting
19:56:03 (49526): No heartbeat from core client for 30 sec - exiting
19:56:04 (49526): No heartbeat from core client for 30 sec - exiting
19:56:05 (49526): No heartbeat from core client for 30 sec - exiting
19:56:06 (49526): No heartbeat from core client for 30 sec - exiting
19:56:07 (49526): No heartbeat from core client for 30 sec - exiting
19:56:08 (49526): No heartbeat from core client for 30 sec - exiting
19:56:09 (49526): No heartbeat from core client for 30 sec - exiting
19:56:10 (49526): No heartbeat from core client for 30 sec - exiting
19:56:11 (49526): No heartbeat from core client for 30 sec - exiting
19:56:12 (49526): No heartbeat from core client for 30 sec - exiting
19:56:13 (49526): No heartbeat from core client for 30 sec - exiting
19:56:14 (49526): No heartbeat from core client for 30 sec - exiting
19:56:15 (49526): No heartbeat from core client for 30 sec - exiting
19:56:16 (49526): No heartbeat from core client for 30 sec - exiting
19:56:17 (49526): No heartbeat from core client for 30 sec - exiting
19:56:18 (49526): No heartbeat from core client for 30 sec - exiting
19:56:19 (49526): No heartbeat from core client for 30 sec - exiting
19:56:20 (49526): No heartbeat from core client for 30 sec - exiting
19:56:21 (49526): No heartbeat from core client for 30 sec - exiting
19:56:22 (49526): No heartbeat from core client for 30 sec - exiting
19:56:23 (49526): No heartbeat from core client for 30 sec - exiting
19:56:24 (49526): No heartbeat from core client for 30 sec - exiting
19:56:25 (49526): No heartbeat from core client for 30 sec - exiting
20:00:34 (49718): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:12:13 (49908): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:03 (50682): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:04 (50682): No heartbeat from core client for 30 sec - exiting
21:17:05 (50682): No heartbeat from core client for 30 sec - exiting
21:17:06 (50682): No heartbeat from core client for 30 sec - exiting
03:14:02 (50869): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
03:18:27 (54294): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:18:28 (54294): No heartbeat from core client for 30 sec - exiting
03:18:29 (54294): No heartbeat from core client for 30 sec - exiting
03:18:30 (54294): No heartbeat from core client for 30 sec - exiting
03:37:01 (54480): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:58:24 (54869): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:03:00 (55280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:07:30 (55468): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:07:31 (55468): No heartbeat from core client for 30 sec - exiting
04:07:32 (55468): No heartbeat from core client for 30 sec - exiting
04:07:33 (55468): No heartbeat from core client for 30 sec - exiting
04:07:34 (55468): No heartbeat from core client for 30 sec - exiting
04:07:35 (55468): No heartbeat from core client for 30 sec - exiting
04:07:36 (55468): No heartbeat from core client for 30 sec - exiting
04:07:37 (55468): No heartbeat from core client for 30 sec - exiting
04:07:38 (55468): No heartbeat from core client for 30 sec - exiting
04:11:40 (55649): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:11:41 (55649): No heartbeat from core client for 30 sec - exiting
04:11:42 (55649): No heartbeat from core client for 30 sec - exiting
04:11:43 (55649): No heartbeat from core client for 30 sec - exiting
04:11:44 (55649): No heartbeat from core client for 30 sec - exiting
06:42:20 (55828): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:45:52 (57306): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:12:07 (57472): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:29:14 (59256): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:33:44 (60654): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:33:45 (60654): No heartbeat from core client for 30 sec - exiting
11:33:46 (60654): No heartbeat from core client for 30 sec - exiting
11:33:47 (60654): No heartbeat from core client for 30 sec - exiting
11:33:48 (60654): No heartbeat from core client for 30 sec - exiting
11:33:49 (60654): No heartbeat from core client for 30 sec - exiting
11:33:50 (60654): No heartbeat from core client for 30 sec - exiting
11:33:51 (60654): No heartbeat from core client for 30 sec - exiting
11:33:52 (60654): No heartbeat from core client for 30 sec - exiting
11:33:53 (60654): No heartbeat from core client for 30 sec - exiting
11:38:10 (60820): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:38:11 (60820): No heartbeat from core client for 30 sec - exiting
11:38:12 (60820): No heartbeat from core client for 30 sec - exiting
11:38:13 (60820): No heartbeat from core client for 30 sec - exiting
11:38:14 (60820): No heartbeat from core client for 30 sec - exiting
11:38:15 (60820): No heartbeat from core client for 30 sec - exiting
11:38:16 (60820): No heartbeat from core client for 30 sec - exiting
11:38:17 (60820): No heartbeat from core client for 30 sec - exiting
11:38:18 (60820): No heartbeat from core client for 30 sec - exiting
11:38:19 (60820): No heartbeat from core client for 30 sec - exiting
11:38:20 (60820): No heartbeat from core client for 30 sec - exiting
11:38:21 (60820): No heartbeat from core client for 30 sec - exiting
11:38:22 (60820): No heartbeat from core client for 30 sec - exiting
11:38:23 (60820): No heartbeat from core client for 30 sec - exiting
11:38:24 (60820): No heartbeat from core client for 30 sec - exiting
11:38:25 (60820): No heartbeat from core client for 30 sec - exiting
11:38:26 (60820): No heartbeat from core client for 30 sec - exiting
MainError:	02:13:34 PM	No files match the supplied pattern.
MainError:	02:13:34 PM	No files match the supplied pattern.
15:37:06 (60994): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:20:28 (63381): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:24:55 (64506): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:09:34 (64658): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:09:35 (64658): No heartbeat from core client for 30 sec - exiting
20:09:36 (64658): No heartbeat from core client for 30 sec - exiting
20:09:37 (64658): No heartbeat from core client for 30 sec - exiting
20:09:38 (64658): No heartbeat from core client for 30 sec - exiting
20:09:39 (64658): No heartbeat from core client for 30 sec - exiting
20:09:40 (64658): No heartbeat from core client for 30 sec - exiting
20:09:41 (64658): No heartbeat from core client for 30 sec - exiting
20:09:42 (64658): No heartbeat from core client for 30 sec - exiting
20:14:05 (1119): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:31:18 (1298): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:31:19 (1298): No heartbeat from core client for 30 sec - exiting
20:35:22 (1604): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:35:23 (1604): No heartbeat from core client for 30 sec - exiting
20:35:24 (1604): No heartbeat from core client for 30 sec - exiting
20:35:25 (1604): No heartbeat from core client for 30 sec - exiting
20:35:26 (1604): No heartbeat from core client for 30 sec - exiting
20:35:27 (1604): No heartbeat from core client for 30 sec - exiting
20:35:28 (1604): No heartbeat from core client for 30 sec - exiting
20:35:29 (1604): No heartbeat from core client for 30 sec - exiting
20:35:30 (1604): No heartbeat from core client for 30 sec - exiting
20:35:31 (1604): No heartbeat from core client for 30 sec - exiting
20:35:32 (1604): No heartbeat from core client for 30 sec - exiting
20:35:33 (1604): No heartbeat from core client for 30 sec - exiting
21:10:09 (1770): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7787400]
[0xf7787425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75a41df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75a7825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf758f4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf77a6400]
[0xf77a6425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75c31df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c6825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75ae4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf76f2400]
[0xf76f2425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf750f1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7512825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74fa4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7742400]
[0xf7742425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf755f1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7562825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf754a4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf774e400]
[0xf774e425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf756b1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf756e825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75564d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf76df400]
[0xf76df425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf74fc1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf74ff825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74e74d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2350, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
31 May 2013 14:59:15 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 570,240 1,518,398 2.6627
30 May 2013 17:58:43 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 544,320 1,453,608 2.6705
29 May 2013 18:56:31 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 518,400 1,378,101 2.6584
28 May 2013 20:11:33 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 492,480 1,300,248 2.6402
27 May 2013 23:30:05 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 466,560 1,228,508 2.6331
27 May 2013 03:32:05 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 440,640 1,161,806 2.6366
26 May 2013 08:09:52 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 414,720 1,094,822 2.6399
25 May 2013 13:23:38 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 388,800 1,027,445 2.6426
24 May 2013 17:24:17 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 362,880 961,238 2.6489
23 May 2013 22:39:21 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 336,960 894,063 2.6533
23 May 2013 03:11:08 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 311,040 826,664 2.6577
22 May 2013 07:02:39 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 285,120 760,097 2.6659
21 May 2013 11:44:08 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 259,200 690,086 2.6624
20 May 2013 16:04:10 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 233,280 619,111 2.6539
19 May 2013 19:44:17 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 207,360 548,688 2.6461
19 May 2013 00:09:02 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 181,440 477,940 2.6341
18 May 2013 03:26:35 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 155,520 407,445 2.6199
17 May 2013 07:09:43 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 129,600 336,909 2.5996
16 May 2013 10:37:56 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 103,680 266,830 2.5736
15 May 2013 14:47:36 1282401 15779612 hadcm3n_o2af_2140_40_008269113_2 77,760 196,530 2.5274


©2024 cpdn.org