climateprediction.net home page
Task 10985832

Task 10985832

Name hadsm3dhet2_jmp4_006592410_5
Workunit 6795783
Created 15 Mar 2010, 11:57:20 UTC
Sent 15 Oct 2010, 10:12:55 UTC
Report deadline 27 Sep 2011, 15:32:55 UTC
Received 28 Jan 2012, 10:09:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 660914
Run time 34 days 1 hours 46 min 40 sec
CPU time 30 days 1 hours 37 min 7 sec
Validate state Invalid
Credit 6,252.33
Device peak FLOPS 1.36 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.08
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.17</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN process is not running, exiting, bRetVal = 1, checkPID=18758, selfPID=18758, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=15666, selfPID=15666, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=17318, selfPID=17318, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=16021, selfPID=16021, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=19061, selfPID=19061, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=3659, selfPID=3659, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=4114, selfPID=4114, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=30752, selfPID=30752, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=30336, selfPID=30336, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=5571, selfPID=5571, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=6814, selfPID=6814, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=6162, selfPID=6162, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
MainError:	07:40:27 AM	No files match the supplied pattern.
CPDN process is not running, exiting, bRetVal = 1, checkPID=1522, selfPID=1522, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=22100, selfPID=22100, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24355, selfPID=24355, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=10042, selfPID=10042, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=17230, selfPID=17230, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=21073, selfPID=21073, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=27133, selfPID=27133, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=5040, selfPID=5040, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=16451, selfPID=16451, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11781, selfPID=11781, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=13390, selfPID=13390, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=14025, selfPID=14025, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=16126, selfPID=16126, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=23077, selfPID=23077, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=28008, selfPID=28008, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=11379, selfPID=11379, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=14121, selfPID=14121, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=23234, selfPID=23234, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=16337, selfPID=16337, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=17635, selfPID=17635, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=29958, selfPID=29958, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=2741, selfPID=2741, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=1940, selfPID=1940, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=8232, selfPID=8232, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=5472, selfPID=5472, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=7205, selfPID=7205, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=5656, selfPID=5656, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=19820, selfPID=19820, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=26346, selfPID=26346, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=24903, selfPID=24903, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=25761, selfPID=25761, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=31437, selfPID=31437, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=3052, selfPID=3052, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=5399, selfPID=5399, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=9097, selfPID=9097, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=16291, selfPID=16291, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=16951, selfPID=16951, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=22068, selfPID=22068, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=28163, selfPID=28163, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=27474, selfPID=27474, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
MainError:	09:06:52 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=2456, selfPID=2456, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=10524, selfPID=10524, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=17171, selfPID=17171, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=17477, selfPID=17477, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=16755, selfPID=16755, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=27469, selfPID=27469, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=29460, selfPID=29460, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=27898, selfPID=27898, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=4860, selfPID=4860, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=5037, selfPID=5037, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=11473, selfPID=11473, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=11840, selfPID=11840, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=15393, selfPID=15393, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=17033, selfPID=17033, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18944, selfPID=18944, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19632, selfPID=19632, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=22450, selfPID=22450, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=23929, selfPID=23929, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=24596, selfPID=24596, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=21019, selfPID=21019, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=26510, selfPID=26510, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=22101, selfPID=22101, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=22389, selfPID=22389, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=30633, selfPID=30633, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=30867, selfPID=30867, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=4294, selfPID=4294, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10050, selfPID=10050, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=6477, selfPID=6477, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=9955, selfPID=9955, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18264, selfPID=18264, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18379, selfPID=18379, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18417, selfPID=18417, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=18517, selfPID=18517, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=19499, selfPID=19499, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=20730, selfPID=20730, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=10812, selfPID=10812, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN process is not running, exiting, bRetVal = 1, checkPID=18141, selfPID=18141, iMonCtr=1
CPDN process is not running, exiting, bRetVal = 1, checkPID=23893, selfPID=23893, iMonCtr=1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: (null)

Model crashed: (null)

Model crashed: (null)

Model crashed: (null)

Model crashed: (null)

Model crashed: (null)
Sorry, too many model crashes! :-(
called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
28 Jan 2012 07:36:41 660914 10985832 hadsm3dhet2_jmp4_006592410_5 162,030 2,589,515 3.8052
26 Jan 2012 09:59:24 660914 10985832 hadsm3dhet2_jmp4_006592410_5 151,228 2,547,436 3.8037
24 Jan 2012 12:47:41 660914 10985832 hadsm3dhet2_jmp4_006592410_5 140,426 2,505,639 3.8026
23 Jan 2012 07:39:57 660914 10985832 hadsm3dhet2_jmp4_006592410_5 129,624 2,463,423 3.8009
21 Jan 2012 09:56:00 660914 10985832 hadsm3dhet2_jmp4_006592410_5 118,822 2,421,373 3.7993
19 Jan 2012 12:47:45 660914 10985832 hadsm3dhet2_jmp4_006592410_5 108,020 2,379,037 3.7972
18 Jan 2012 07:48:58 660914 10985832 hadsm3dhet2_jmp4_006592410_5 97,218 2,336,909 3.7954
16 Jan 2012 10:55:11 660914 10985832 hadsm3dhet2_jmp4_006592410_5 86,416 2,294,832 3.7937
15 Jan 2012 06:14:54 660914 10985832 hadsm3dhet2_jmp4_006592410_5 75,614 2,252,437 3.7913
13 Jan 2012 09:32:49 660914 10985832 hadsm3dhet2_jmp4_006592410_5 64,812 2,210,675 3.7899
11 Jan 2012 12:24:57 660914 10985832 hadsm3dhet2_jmp4_006592410_5 54,010 2,168,775 3.7882
10 Jan 2012 06:32:50 660914 10985832 hadsm3dhet2_jmp4_006592410_5 43,208 2,126,546 3.7859
08 Jan 2012 08:44:51 660914 10985832 hadsm3dhet2_jmp4_006592410_5 32,406 2,085,492 3.7856
06 Jan 2012 11:13:34 660914 10985832 hadsm3dhet2_jmp4_006592410_5 21,604 2,043,516 3.7836
04 Jan 2012 13:57:24 660914 10985832 hadsm3dhet2_jmp4_006592410_5 10,802 2,001,794 3.7820
03 Jan 2012 09:11:22 660914 10985832 hadsm3dhet2_jmp4_006592410_5 259,248 1,959,540 3.7793
01 Jan 2012 11:42:53 660914 10985832 hadsm3dhet2_jmp4_006592410_5 248,446 1,917,683 3.7772
31 Dec 2011 06:25:53 660914 10985832 hadsm3dhet2_jmp4_006592410_5 237,644 1,875,417 3.7743
29 Dec 2011 08:58:59 660914 10985832 hadsm3dhet2_jmp4_006592410_5 226,842 1,833,528 3.7720
27 Dec 2011 11:38:10 660914 10985832 hadsm3dhet2_jmp4_006592410_5 216,040 1,791,335 3.7689


©2024 climateprediction.net