climateprediction.net home page
Task 11900785

Task 11900785

Name hadsm3dhet2_u4gt_006726036_9
Workunit 6929379
Created 17 Sep 2010, 8:08:54 UTC
Sent 20 Sep 2010, 11:34:43 UTC
Report deadline 2 Sep 2011, 16:54:43 UTC
Received 13 Dec 2010, 0:04:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1266644
Run time 14 days 20 hours 21 min 15 sec
CPU time 14 days 12 hours 46 min 6 sec
Validate state Invalid
Credit 4,862.93
Device peak FLOPS 1.44 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.08
i686-pc-linux-gnu
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19155, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=19987, selfPID=19987, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20172, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=20245, selfPID=20245, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10625, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=10839, selfPID=10839, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=6441, selfPID=6441, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=7603, selfPID=7603, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=8446, selfPID=8446, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=13087, selfPID=13087, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13087, selfPID=13087, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13087, selfPID=13087, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13087, selfPID=13087, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13087, selfPID=13087, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=14571, selfPID=14571, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14571, selfPID=14571, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14836, selfPID=14836, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=16930, selfPID=16930, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=18465, selfPID=18465, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18465, selfPID=18465, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18465, selfPID=18465, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18465, selfPID=18465, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18962, selfPID=18962, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=19625, selfPID=19625, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=20203, selfPID=20203, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=20314, selfPID=20314, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=22639, selfPID=22639, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=22639, selfPID=22639, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24075, selfPID=24075, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24075, selfPID=24075, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=10596, selfPID=10596, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10596, selfPID=10596, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10596, selfPID=10596, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10596, selfPID=10596, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10596, selfPID=10596, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11223, selfPID=11223, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=13452, selfPID=13452, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=13815, selfPID=13815, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=14397, selfPID=14397, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14397, selfPID=14397, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=15436, selfPID=15436, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=18179, selfPID=18179, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18400, selfPID=18400, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18425, selfPID=18425, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18618, selfPID=18618, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18618, selfPID=18618, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18708, selfPID=18708, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19290, selfPID=19290, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19458, selfPID=19458, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19458, selfPID=19458, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19613, selfPID=19613, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=20039, selfPID=20039, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=20039, selfPID=20039, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=20283, selfPID=20283, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=22452, selfPID=22452, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=23842, selfPID=23842, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=23842, selfPID=23842, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=25595, selfPID=25595, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25595, selfPID=25595, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25595, selfPID=25595, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25595, selfPID=25595, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25595, selfPID=25595, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=28274, selfPID=28274, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=28537, selfPID=28537, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=28800, selfPID=28800, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29045, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=29341, selfPID=29341, iMonCtr=1
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
MainError:	03:00:37 PM	No files match the supplied pattern.
CPDN process is not running, exiting, bRetVal = 1, checkPID=14486, selfPID=14486, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=21644, selfPID=21644, iMonCtr=1
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
MainError:	03:28:28 PM	No files match the supplied pattern.
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
17 Oct 2010 03:34:58 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 10,802 1,231,499 2.3267
16 Oct 2010 15:31:30 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 259,248 1,205,748 2.3255
10 Oct 2010 04:16:24 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 248,446 1,182,263 2.3287
10 Oct 2010 04:16:24 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 237,644 1,159,468 2.3334
10 Oct 2010 04:16:24 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 226,842 1,136,869 2.3388
10 Oct 2010 04:16:23 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 216,040 1,113,869 2.3436
10 Oct 2010 04:16:23 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 205,238 1,090,667 2.3481
10 Oct 2010 04:16:23 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 194,436 1,067,830 2.3537
10 Oct 2010 04:16:23 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 183,634 1,044,776 2.3590
10 Oct 2010 04:16:23 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 172,832 1,021,708 2.3646
06 Oct 2010 11:24:08 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 162,030 997,581 2.3680
06 Oct 2010 04:40:18 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 151,228 973,450 2.3715
05 Oct 2010 22:17:54 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 140,426 950,787 2.3789
05 Oct 2010 15:48:09 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 129,624 927,488 2.3851
04 Oct 2010 06:36:55 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 118,822 902,399 2.3869
03 Oct 2010 22:31:11 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 108,020 876,661 2.3870
03 Oct 2010 10:12:56 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 97,218 852,181 2.3906
03 Oct 2010 03:23:07 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 86,416 826,720 2.3917
02 Oct 2010 20:11:52 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 75,614 802,059 2.3952
02 Oct 2010 13:12:30 1099403 11900785 hadsm3dhet2_u4gt_006726036_9 64,812 776,785 2.3970


©2024 climateprediction.net