climateprediction.net home page
Task 11900794

Task 11900794

Name hadsm3dhet2_u5ea_006726037_8
Workunit 6929380
Created 17 Sep 2010, 8:08:54 UTC
Sent 20 Sep 2010, 11:34:43 UTC
Report deadline 2 Sep 2011, 16:54:43 UTC
Received 13 Dec 2010, 0:04:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1266644
Run time 17 days 19 hours 40 min 36 sec
CPU time 17 days 11 hours 28 min 9 sec
Validate state Invalid
Credit 5,954.60
Device peak FLOPS 1.44 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.08
i686-pc-linux-gnu
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=19151, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=19988, selfPID=19988, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20169, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=20249, selfPID=20249, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10624, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=10841, selfPID=10841, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=5680, selfPID=5680, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=5680, selfPID=5680, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=6440, selfPID=6440, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=8441, selfPID=8441, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=9041, selfPID=9041, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=9041, selfPID=9041, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=9041, selfPID=9041, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=9657, selfPID=9657, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11941, selfPID=11941, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=12450, selfPID=12450, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=12948, selfPID=12948, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13085, selfPID=13085, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13085, selfPID=13085, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14210, selfPID=14210, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14210, selfPID=14210, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=14572, selfPID=14572, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=19623, selfPID=19623, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19623, selfPID=19623, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=20671, selfPID=20671, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=21275, selfPID=21275, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=21967, selfPID=21967, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=22637, selfPID=22637, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24068, selfPID=24068, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=1779, selfPID=1779, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=2114, selfPID=2114, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=2777, selfPID=2777, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=4110, selfPID=4110, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=4110, selfPID=4110, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=7332, selfPID=7332, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=8297, selfPID=8297, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=10184, selfPID=10184, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=10605, selfPID=10605, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11222, selfPID=11222, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11861, selfPID=11861, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=13447, selfPID=13447, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14149, selfPID=14149, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=16255, selfPID=16255, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=17834, selfPID=17834, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=18178, selfPID=18178, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18395, selfPID=18395, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18395, selfPID=18395, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=18619, selfPID=18619, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=19453, selfPID=19453, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19453, selfPID=19453, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=21611, selfPID=21611, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=23831, selfPID=23831, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24146, selfPID=24146, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24146, selfPID=24146, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=24434, selfPID=24434, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24434, selfPID=24434, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24434, selfPID=24434, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=25600, selfPID=25600, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25600, selfPID=25600, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25600, selfPID=25600, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=28527, selfPID=28527, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=28794, selfPID=28794, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=29040, iMonCtr=1
Model crash detected, will try to restart...
CPDN process is not running, exiting, bRetVal = 1, checkPID=29338, selfPID=29338, iMonCtr=1
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
MainError:	08:20:09 AM	No files match the supplied pattern.
CPDN process is not running, exiting, bRetVal = 1, checkPID=30458, selfPID=30458, iMonCtr=1
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
MainError:	04:54:17 PM	No files match the supplied pattern.
CPDN process is not running, exiting, bRetVal = 1, checkPID=27467, selfPID=27467, iMonCtr=1
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
/scratch/.boinc.beo-33/projects/climateprediction.net/hadsm3_um_6.08_i686-pc-linux-gnu: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64
CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=47017, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
03 Dec 2010 09:23:52 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 129,624 1,488,366 2.2964
03 Dec 2010 09:10:15 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 118,822 1,465,913 2.3001
02 Dec 2010 20:00:21 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 108,020 1,443,373 2.3038
02 Dec 2010 13:49:04 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 97,218 1,421,080 2.3080
02 Dec 2010 07:43:10 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 86,416 1,399,021 2.3128
02 Dec 2010 07:28:49 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 75,614 1,376,804 2.3174
01 Dec 2010 19:02:53 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 64,812 1,354,570 2.3222
01 Dec 2010 12:51:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 54,010 1,332,244 2.3270
01 Dec 2010 08:48:31 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 43,208 1,309,880 2.3320
25 Oct 2010 13:43:11 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 32,406 1,287,315 2.3367
17 Oct 2010 03:34:58 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 21,604 1,261,563 2.3358
16 Oct 2010 15:00:06 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 10,802 1,235,891 2.3350
10 Oct 2010 04:16:24 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 259,248 1,212,249 2.3380
10 Oct 2010 04:16:24 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 248,446 1,189,457 2.3429
10 Oct 2010 04:16:24 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 237,644 1,166,584 2.3478
10 Oct 2010 04:16:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 226,842 1,144,256 2.3540
10 Oct 2010 04:16:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 216,040 1,121,351 2.3593
10 Oct 2010 04:16:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 205,238 1,097,773 2.3634
10 Oct 2010 04:16:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 194,436 1,074,315 2.3680
10 Oct 2010 04:16:23 1099403 11900794 hadsm3dhet2_u5ea_006726037_8 183,634 1,050,941 2.3730


©2024 climateprediction.net