climateprediction.net home page
Task 15839735

Task 15839735

Name hadcm3n_48vu_1980_40_008365028_4
Workunit 8515887
Created 12 Jun 2013, 3:05:08 UTC
Sent 12 Jun 2013, 3:27:49 UTC
Report deadline 11 Sep 2013, 10:55:00 UTC
Received 18 Jul 2013, 11:58:27 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1113142
Run time 24 days 13 hours 46 min 13 sec
CPU time 17 days 12 hours 52 min 36 sec
Validate state Invalid
Credit 10,264.32
Device peak FLOPS 2.59 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
The device does not recognize the command. (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
15:40:27 (6248): No heartbeat from core client for 30 sec - exiting
15:40:28 (6248): No heartbeat from core client for 30 sec - exiting
15:40:29 (6248): No heartbeat from core client for 30 sec - exiting
15:40:30 (6248): No heartbeat from core client for 30 sec - exiting
15:40:31 (6248): No heartbeat from core client for 30 sec - exiting
15:40:32 (6248): No heartbeat from core client for 30 sec - exiting
15:40:33 (6248): No heartbeat from core client for 30 sec - exiting
15:40:34 (6248): No heartbeat from core client for 30 sec - exiting
15:40:35 (6248): No heartbeat from core client for 30 sec - exiting
15:40:36 (6248): No heartbeat from core client for 30 sec - exiting
15:40:38 (6248): No heartbeat from core client for 30 sec - exiting
15:40:39 (6248): No heartbeat from core client for 30 sec - exiting
15:40:40 (6248): No heartbeat from core client for 30 sec - exiting
15:40:41 (6248): No heartbeat from core client for 30 sec - exiting
15:40:42 (6248): No heartbeat from core client for 30 sec - exiting
15:40:43 (6248): No heartbeat from core client for 30 sec - exiting
15:40:44 (6248): No heartbeat from core client for 30 sec - exiting
15:40:45 (6248): No heartbeat from core client for 30 sec - exiting
15:40:46 (6248): No heartbeat from core client for 30 sec - exiting
15:40:47 (6248): No heartbeat from core client for 30 sec - exiting
15:40:48 (6248): No heartbeat from core client for 30 sec - exiting
15:40:50 (6248): No heartbeat from core client for 30 sec - exiting
15:40:51 (6248): No heartbeat from core client for 30 sec - exiting
15:40:52 (6248): No heartbeat from core client for 30 sec - exiting
15:40:53 (6248): No heartbeat from core client for 30 sec - exiting
15:40:54 (6248): No heartbeat from core client for 30 sec - exiting
15:40:55 (6248): No heartbeat from core client for 30 sec - exiting
15:40:56 (6248): No heartbeat from core client for 30 sec - exiting
15:40:57 (6248): No heartbeat from core client for 30 sec - exiting
15:40:58 (6248): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6700, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
21:07:50 (7692): No heartbeat from core client for 30 sec - exiting
21:07:51 (7692): No heartbeat from core client for 30 sec - exiting
21:07:52 (7692): No heartbeat from core client for 30 sec - exiting
21:07:53 (7692): No heartbeat from core client for 30 sec - exiting
21:07:54 (7692): No heartbeat from core client for 30 sec - exiting
21:07:55 (7692): No heartbeat from core client for 30 sec - exiting
21:07:56 (7692): No heartbeat from core client for 30 sec - exiting
21:07:57 (7692): No heartbeat from core client for 30 sec - exiting
21:07:58 (7692): No heartbeat from core client for 30 sec - exiting
21:08:00 (7692): No heartbeat from core client for 30 sec - exiting
21:08:01 (7692): No heartbeat from core client for 30 sec - exiting
21:08:02 (7692): No heartbeat from core client for 30 sec - exiting
21:08:03 (7692): No heartbeat from core client for 30 sec - exiting
21:08:04 (7692): No heartbeat from core client for 30 sec - exiting
21:08:05 (7692): No heartbeat from core client for 30 sec - exiting
21:08:06 (7692): No heartbeat from core client for 30 sec - exiting
21:08:07 (7692): No heartbeat from core client for 30 sec - exiting
21:08:08 (7692): No heartbeat from core client for 30 sec - exiting
21:08:09 (7692): No heartbeat from core client for 30 sec - exiting
21:08:10 (7692): No heartbeat from core client for 30 sec - exiting
21:08:12 (7692): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:41 (10900): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
17:44:10 (4932): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
Suspended CPDN Monitor - Suspend request from BOINC...
18:16:49 (16292): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:44:51 (7448): No heartbeat from core client for 30 sec - exiting
14:44:52 (7448): No heartbeat from core client for 30 sec - exiting
14:44:53 (7448): No heartbeat from core client for 30 sec - exiting
14:44:54 (7448): No heartbeat from core client for 30 sec - exiting
14:44:55 (7448): No heartbeat from core client for 30 sec - exiting
14:44:56 (7448): No heartbeat from core client for 30 sec - exiting
14:44:58 (7448): No heartbeat from core client for 30 sec - exiting
14:44:59 (7448): No heartbeat from core client for 30 sec - exiting
14:45:00 (7448): No heartbeat from core client for 30 sec - exiting
14:45:01 (7448): No heartbeat from core client for 30 sec - exiting
14:45:02 (7448): No heartbeat from core client for 30 sec - exiting
14:45:03 (7448): No heartbeat from core client for 30 sec - exiting
14:45:04 (7448): No heartbeat from core client for 30 sec - exiting
14:45:05 (7448): No heartbeat from core client for 30 sec - exiting
14:45:06 (7448): No heartbeat from core client for 30 sec - exiting
14:45:07 (7448): No heartbeat from core client for 30 sec - exiting
14:45:08 (7448): No heartbeat from core client for 30 sec - exiting
14:45:10 (7448): No heartbeat from core client for 30 sec - exiting
14:45:11 (7448): No heartbeat from core client for 30 sec - exiting
14:45:12 (7448): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:16:05 (9660): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:16:16 (9660): No heartbeat from core client for 30 sec - exiting
16:16:17 (9660): No heartbeat from core client for 30 sec - exiting
16:16:18 (9660): No heartbeat from core client for 30 sec - exiting
16:16:19 (9660): No heartbeat from core client for 30 sec - exiting
16:16:20 (9660): No heartbeat from core client for 30 sec - exiting
16:16:21 (9660): No heartbeat from core client for 30 sec - exiting
16:16:22 (9660): No heartbeat from core client for 30 sec - exiting
16:16:23 (9660): No heartbeat from core client for 30 sec - exiting
16:16:24 (9660): No heartbeat from core client for 30 sec - exiting
16:16:25 (9660): No heartbeat from core client for 30 sec - exiting
16:17:07 (3972): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
15:34:42 (5184): No heartbeat from core client for 30 sec - exiting
15:34:43 (5184): No heartbeat from core client for 30 sec - exiting
15:34:44 (5184): No heartbeat from core client for 30 sec - exiting
15:34:45 (5184): No heartbeat from core client for 30 sec - exiting
15:34:46 (5184): No heartbeat from core client for 30 sec - exiting
15:34:47 (5184): No heartbeat from core client for 30 sec - exiting
15:34:48 (5184): No heartbeat from core client for 30 sec - exiting
15:34:50 (5184): No heartbeat from core client for 30 sec - exiting
15:34:51 (5184): No heartbeat from core client for 30 sec - exiting
15:34:52 (5184): No heartbeat from core client for 30 sec - exiting
15:34:53 (5184): No heartbeat from core client for 30 sec - exiting
15:34:54 (5184): No heartbeat from core client for 30 sec - exiting
15:34:55 (5184): No heartbeat from core client for 30 sec - exiting
15:34:56 (5184): No heartbeat from core client for 30 sec - exiting
15:34:57 (5184): No heartbeat from core client for 30 sec - exiting
15:34:58 (5184): No heartbeat from core client for 30 sec - exiting
15:34:59 (5184): No heartbeat from core client for 30 sec - exiting
15:35:00 (5184): No heartbeat from core client for 30 sec - exiting
15:35:02 (5184): No heartbeat from core client for 30 sec - exiting
15:35:03 (5184): No heartbeat from core client for 30 sec - exiting
15:35:04 (5184): No heartbeat from core client for 30 sec - exiting
15:35:05 (5184): No heartbeat from core client for 30 sec - exiting
15:35:06 (5184): No heartbeat from core client for 30 sec - exiting
15:35:07 (5184): No heartbeat from core client for 30 sec - exiting
15:35:08 (5184): No heartbeat from core client for 30 sec - exiting
15:35:09 (5184): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
04:06:24 (6572): No heartbeat from core client for 30 sec - exiting
04:06:25 (6572): No heartbeat from core client for 30 sec - exiting
04:06:26 (6572): No heartbeat from core client for 30 sec - exiting
04:06:27 (6572): No heartbeat from core client for 30 sec - exiting
04:06:28 (6572): No heartbeat from core client for 30 sec - exiting
04:06:29 (6572): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:06:30 (6572): No heartbeat from core client for 30 sec - exiting
11:50:02 (9832): No heartbeat from core client for 30 sec - exiting
11:50:03 (9832): No heartbeat from core client for 30 sec - exiting
11:50:04 (9832): No heartbeat from core client for 30 sec - exiting
11:50:05 (9832): No heartbeat from core client for 30 sec - exiting
11:50:07 (9832): No heartbeat from core client for 30 sec - exiting
11:50:08 (9832): No heartbeat from core client for 30 sec - exiting
11:50:09 (9832): No heartbeat from core client for 30 sec - exiting
11:50:10 (9832): No heartbeat from core client for 30 sec - exiting
11:50:11 (9832): No heartbeat from core client for 30 sec - exiting
11:50:12 (9832): No heartbeat from core client for 30 sec - exiting
11:50:13 (9832): No heartbeat from core client for 30 sec - exiting
11:50:14 (9832): No heartbeat from core client for 30 sec - exiting
11:50:15 (9832): No heartbeat from core client for 30 sec - exiting
11:50:16 (9832): No heartbeat from core client for 30 sec - exiting
11:50:18 (9832): No heartbeat from core client for 30 sec - exiting
11:50:19 (9832): No heartbeat from core client for 30 sec - exiting
11:50:20 (9832): No heartbeat from core client for 30 sec - exiting
11:50:21 (9832): No heartbeat from core client for 30 sec - exiting
11:50:22 (9832): No heartbeat from core client for 30 sec - exiting
11:50:23 (9832): No heartbeat from core client for 30 sec - exiting
11:50:24 (9832): No heartbeat from core client for 30 sec - exiting
11:50:25 (9832): No heartbeat from core client for 30 sec - exiting
11:50:26 (9832): No heartbeat from core client for 30 sec - exiting
11:50:27 (9832): No heartbeat from core client for 30 sec - exiting
11:50:28 (9832): No heartbeat from core client for 30 sec - exiting
11:50:30 (9832): No heartbeat from core client for 30 sec - exiting
11:50:31 (9832): No heartbeat from core client for 30 sec - exiting
11:50:32 (9832): No heartbeat from core client for 30 sec - exiting
11:50:33 (9832): No heartbeat from core client for 30 sec - exiting
11:50:34 (9832): No heartbeat from core client for 30 sec - exiting
11:50:35 (9832): No heartbeat from core client for 30 sec - exiting
11:50:36 (9832): No heartbeat from core client for 30 sec - exiting
11:50:37 (9832): No heartbeat from core client for 30 sec - exiting
11:50:38 (9832): No heartbeat from core client for 30 sec - exiting
11:50:39 (9832): No heartbeat from core client for 30 sec - exiting
11:50:40 (9832): No heartbeat from core client for 30 sec - exiting
11:50:42 (9832): No heartbeat from core client for 30 sec - exiting
11:50:43 (9832): No heartbeat from core client for 30 sec - exiting
11:50:44 (9832): No heartbeat from core client for 30 sec - exiting
11:50:45 (9832): No heartbeat from core client for 30 sec - exiting
11:50:46 (9832): No heartbeat from core client for 30 sec - exiting
11:50:47 (9832): No heartbeat from core client for 30 sec - exiting
11:50:48 (9832): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:44:11 (11160): No heartbeat from core client for 30 sec - exiting
06:44:26 (11160): No heartbeat from core client for 30 sec - exiting
06:44:28 (11160): No heartbeat from core client for 30 sec - exiting
06:44:29 (11160): No heartbeat from core client for 30 sec - exiting
06:44:30 (11160): No heartbeat from core client for 30 sec - exiting
06:44:31 (11160): No heartbeat from core client for 30 sec - exiting
06:44:32 (11160): No heartbeat from core client for 30 sec - exiting
06:44:33 (11160): No heartbeat from core client for 30 sec - exiting
06:44:34 (11160): No heartbeat from core client for 30 sec - exiting
06:44:35 (11160): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:28:19 (12280): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
16:55:36 (4632): No heartbeat from core client for 30 sec - exiting
16:55:37 (4632): No heartbeat from core client for 30 sec - exiting
16:55:38 (4632): No heartbeat from core client for 30 sec - exiting
16:55:39 (4632): No heartbeat from core client for 30 sec - exiting
16:55:40 (4632): No heartbeat from core client for 30 sec - exiting
16:55:41 (4632): No heartbeat from core client for 30 sec - exiting
16:55:42 (4632): No heartbeat from core client for 30 sec - exiting
16:55:43 (4632): No heartbeat from core client for 30 sec - exiting
16:55:44 (4632): No heartbeat from core client for 30 sec - exiting
16:55:46 (4632): No heartbeat from core client for 30 sec - exiting
16:55:47 (4632): No heartbeat from core client for 30 sec - exiting
16:55:48 (4632): No heartbeat from core client for 30 sec - exiting
16:55:49 (4632): No heartbeat from core client for 30 sec - exiting
16:55:50 (4632): No heartbeat from core client for 30 sec - exiting
16:55:51 (4632): No heartbeat from core client for 30 sec - exiting
16:55:52 (4632): No heartbeat from core client for 30 sec - exiting
16:55:53 (4632): No heartbeat from core client for 30 sec - exiting
16:55:54 (4632): No heartbeat from core client for 30 sec - exiting
16:55:55 (4632): No heartbeat from core client for 30 sec - exiting
16:55:56 (4632): No heartbeat from core client for 30 sec - exiting
16:55:58 (4632): No heartbeat from core client for 30 sec - exiting
16:55:59 (4632): No heartbeat from core client for 30 sec - exiting
16:56:00 (4632): No heartbeat from core client for 30 sec - exiting
16:56:01 (4632): No heartbeat from core client for 30 sec - exiting
16:56:02 (4632): No heartbeat from core client for 30 sec - exiting
16:56:03 (4632): No heartbeat from core client for 30 sec - exiting
16:56:04 (4632): No heartbeat from core client for 30 sec - exiting
16:56:05 (4632): No heartbeat from core client for 30 sec - exiting
16:56:06 (4632): No heartbeat from core client for 30 sec - exiting
16:56:07 (4632): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Atmos Hold Restart file rename failed on atmos_restart.hold
18:58:45 (11024): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:58:46 (11024): No heartbeat from core client for 30 sec - exiting
18:58:47 (11024): No heartbeat from core client for 30 sec - exiting
18:58:48 (11024): No heartbeat from core client for 30 sec - exiting
18:58:49 (11024): No heartbeat from core client for 30 sec - exiting
18:58:50 (11024): No heartbeat from core client for 30 sec - exiting
18:58:51 (11024): No heartbeat from core client for 30 sec - exiting
18:58:52 (11024): No heartbeat from core client for 30 sec - exiting
18:58:53 (11024): No heartbeat from core client for 30 sec - exiting
18:58:55 (11024): No heartbeat from core client for 30 sec - exiting
18:58:56 (11024): No heartbeat from core client for 30 sec - exiting
19:37:16 (6672): No heartbeat from core client for 30 sec - exiting
19:37:18 (6672): No heartbeat from core client for 30 sec - exiting
19:37:45 (6672): No heartbeat from core client for 30 sec - exiting
19:37:46 (6672): No heartbeat from core client for 30 sec - exiting
19:37:48 (6672): No heartbeat from core client for 30 sec - exiting
19:37:49 (6672): No heartbeat from core client for 30 sec - exiting
19:37:50 (6672): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:57:27 (8276): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
03:57:28 (8276): No heartbeat from core client for 30 sec - exiting
03:57:29 (8276): No heartbeat from core client for 30 sec - exiting
03:57:30 (8276): No heartbeat from core client for 30 sec - exiting
03:57:31 (8276): No heartbeat from core client for 30 sec - exiting
03:57:32 (8276): No heartbeat from core client for 30 sec - exiting
03:57:33 (8276): No heartbeat from core client for 30 sec - exiting
03:57:34 (8276): No heartbeat from core client for 30 sec - exiting
03:57:35 (8276): No heartbeat from core client for 30 sec - exiting
03:57:36 (8276): No heartbeat from core client for 30 sec - exiting
03:57:37 (8276): No heartbeat from core client for 30 sec - exiting
11:50:40 (11136): No heartbeat from core client for 30 sec - exiting
11:50:42 (11136): No heartbeat from core client for 30 sec - exiting
11:50:43 (11136): No heartbeat from core client for 30 sec - exiting
11:50:44 (11136): No heartbeat from core client for 30 sec - exiting
11:50:45 (11136): No heartbeat from core client for 30 sec - exiting
11:50:46 (11136): No heartbeat from core client for 30 sec - exiting
11:50:47 (11136): No heartbeat from core client for 30 sec - exiting
11:50:48 (11136): No heartbeat from core client for 30 sec - exiting
11:50:49 (11136): No heartbeat from core client for 30 sec - exiting
11:50:50 (11136): No heartbeat from core client for 30 sec - exiting
11:50:51 (11136): No heartbeat from core client for 30 sec - exiting
11:50:53 (11136): No heartbeat from core client for 30 sec - exiting
11:50:54 (11136): No heartbeat from core client for 30 sec - exiting
11:50:55 (11136): No heartbeat from core client for 30 sec - exiting
11:50:56 (11136): No heartbeat from core client for 30 sec - exiting
11:50:57 (11136): No heartbeat from core client for 30 sec - exiting
11:50:58 (11136): No heartbeat from core client for 30 sec - exiting
11:50:59 (11136): No heartbeat from core client for 30 sec - exiting
11:51:00 (11136): No heartbeat from core client for 30 sec - exiting
11:51:01 (11136): No heartbeat from core client for 30 sec - exiting
11:51:02 (11136): No heartbeat from core client for 30 sec - exiting
11:51:03 (11136): No heartbeat from core client for 30 sec - exiting
11:51:05 (11136): No heartbeat from core client for 30 sec - exiting
11:51:06 (11136): No heartbeat from core client for 30 sec - exiting
11:51:07 (11136): No heartbeat from core client for 30 sec - exiting
11:51:08 (11136): No heartbeat from core client for 30 sec - exiting
11:51:09 (11136): No heartbeat from core client for 30 sec - exiting
11:51:10 (11136): No heartbeat from core client for 30 sec - exiting
11:51:11 (11136): No heartbeat from core client for 30 sec - exiting
11:51:12 (11136): No heartbeat from core client for 30 sec - exiting
11:51:13 (11136): No heartbeat from core client for 30 sec - exiting
11:51:14 (11136): No heartbeat from core client for 30 sec - exiting
11:51:15 (11136): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Signal 22 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10936, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Jul 2013 15:28:28 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 855,360 1,489,065 1.7409
23 Jul 2013 15:28:28 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 829,440 1,443,772 1.7407
23 Jul 2013 15:28:28 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 803,520 1,398,927 1.7410
23 Jul 2013 15:28:28 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 777,600 1,353,910 1.7411
11 Jul 2013 14:37:48 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 751,680 1,307,697 1.7397
10 Jul 2013 06:50:29 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 725,760 1,262,609 1.7397
09 Jul 2013 06:40:58 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 699,840 1,217,774 1.7401
08 Jul 2013 06:41:53 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 673,920 1,172,649 1.7400
07 Jul 2013 11:06:30 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 648,000 1,127,167 1.7395
06 Jul 2013 20:19:30 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 622,080 1,082,568 1.7402
06 Jul 2013 05:48:55 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 596,160 1,037,145 1.7397
06 Jul 2013 05:05:31 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 570,240 992,844 1.7411
04 Jul 2013 19:12:46 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 544,320 947,076 1.7399
04 Jul 2013 14:04:57 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 518,400 901,524 1.7391
03 Jul 2013 07:15:01 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 492,480 854,721 1.7355
02 Jul 2013 12:08:28 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 466,560 808,891 1.7337
02 Jul 2013 11:23:01 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 440,640 766,721 1.7400
02 Jul 2013 10:41:20 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 414,720 719,634 1.7352
02 Jul 2013 10:16:34 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 388,800 675,149 1.7365
02 Jul 2013 09:57:25 1113142 15839735 hadcm3n_48vu_1980_40_008365028_4 362,880 629,662 1.7352


©2024 cpdn.org