climateprediction.net home page
Task 13159205

Task 13159205

Name hadcm3n_yk2j_1900_40_007358869_2
Workunit 7556299
Created 26 Jul 2011, 11:39:01 UTC
Sent 26 Jul 2011, 11:39:08 UTC
Report deadline 25 Oct 2011, 19:06:19 UTC
Received 11 Aug 2011, 2:08:44 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1244925
Run time 13 days 14 hours 1 min 39 sec
CPU time 11 days 6 hours 21 min 6 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 3.74 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr
<core_client_version>6.12.26</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 62 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Error: Input file: dataout/yk2jko.pjb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pjb0c10
Error: Input file: dataout/yk2jko.pib0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pib0c10
Error: Input file: dataout/yk2jko.pfb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pfb0c10
Error: Input file: dataout/yk2jko.pcb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pcb0c10
Error: Input file: dataout/yk2jko.pbb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pbb0c10
Error: Input file: dataout/yk2jko.pab0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pab0c10
Error: Input file: dataout/yk2jka.phb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.phb0c10
Error: Input file: dataout/yk2jka.pgb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pgb0c10
Error: Input file: dataout/yk2jka.peb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.peb0c10
Error: Input file: dataout/yk2jka.pdb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pdb0c10
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 62 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Error: Input file: dataout/yk2jko.pjb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pjb0c10
Error: Input file: dataout/yk2jko.pib0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pib0c10
Error: Input file: dataout/yk2jko.pfb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pfb0c10
Error: Input file: dataout/yk2jko.pcb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pcb0c10
Error: Input file: dataout/yk2jko.pbb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pbb0c10
Error: Input file: dataout/yk2jko.pab0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pab0c10
Error: Input file: dataout/yk2jka.phb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.phb0c10
Error: Input file: dataout/yk2jka.pgb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pgb0c10
Error: Input file: dataout/yk2jka.peb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.peb0c10
Error: Input file: dataout/yk2jka.pdb0c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pdb0c10
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
Error: Input file: dataout/yk2jko.pjb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pjb8c10
Error: Input file: dataout/yk2jko.pib8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pib8c10
Error: Input file: dataout/yk2jko.pfb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jko.pfb8c10
Error: Input file: dataout/yk2jka.phb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.phb8c10
Error: Input file: dataout/yk2jka.pgb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pgb8c10
Error: Input file: dataout/yk2jka.peb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.peb8c10
Error: Input file: dataout/yk2jka.pdb8c10 is not a valid UM file.
Error converting file to netcdf: dataout/yk2jka.pdb8c10
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x4020a04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x301f604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x303ac04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x303ac00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x303ac04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x303ac00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(18017,0xa0564540) malloc: *** error for object 0x301f604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=18026, selfPID=18026, iMonCtr=1
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x843404: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x282c604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x843404: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x822604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x822600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(25322,0xa0564540) malloc: *** error for object 0x2806200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_yk2j_1900_40_007358869/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Aug 2011 01:10:26 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 777,600 973,287 1.2517
10 Aug 2011 14:00:28 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 751,680 940,596 1.2513
10 Aug 2011 03:30:58 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 725,760 907,789 1.2508
09 Aug 2011 18:10:58 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 699,840 875,084 1.2504
09 Aug 2011 05:59:38 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 673,920 841,653 1.2489
08 Aug 2011 08:01:56 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 648,000 807,214 1.2457
07 Aug 2011 21:17:01 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 622,080 774,551 1.2451
07 Aug 2011 10:30:14 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 596,160 742,079 1.2448
07 Aug 2011 00:39:08 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 570,240 709,445 1.2441
06 Aug 2011 12:16:20 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 544,320 676,759 1.2433
06 Aug 2011 01:44:24 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 518,400 644,217 1.2427
05 Aug 2011 14:22:35 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 492,480 612,556 1.2438
05 Aug 2011 01:52:07 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 466,560 579,955 1.2430
04 Aug 2011 08:09:01 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 440,640 547,503 1.2425
03 Aug 2011 16:25:01 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 414,720 514,988 1.2418
03 Aug 2011 03:25:50 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 388,800 483,964 1.2448
02 Aug 2011 10:24:30 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 362,880 452,102 1.2459
01 Aug 2011 19:00:48 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 336,960 419,880 1.2461
01 Aug 2011 05:04:03 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 311,040 387,466 1.2457
31 Jul 2011 16:29:21 998485 13159205 hadcm3n_yk2j_1900_40_007358869_2 285,120 354,771 1.2443


©2024 cpdn.org