climateprediction.net home page
Task 11724074

Task 11724074

Name famous_v5h3_1199_200_006692861_0
Workunit 6896114
Created 26 Aug 2010, 16:01:45 UTC
Sent 9 Sep 2010, 3:06:27 UTC
Report deadline 9 Dec 2010, 10:33:38 UTC
Received 22 Sep 2010, 20:54:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 922557
Run time
CPU time 5 days 12 hours 35 min 28 sec
Validate state Invalid
Credit 1,976.51
Device peak FLOPS 1.69 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>5.10.45</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
_v5h3_1199_200_006692861_0_1256.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0533521 A - 01/12/1256 01:00 - H:M:S=0115:17:51 AVG= 0.78 DLT= 1.98

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 11857
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_v5h3_1199_200_006692861 - PH 1 TS 0533911 A - 16/12/1256 01:00 - H:M:S=0115:26:34 AVG= 0.78 DLT= 1.67
famous_v5h3_1199_200_006692861 - PH 1 TS 0534301 A - 01/01/1257 01:00 - H:M:S=0115:31:01 AVG= 0.78 DLT= 1.39
famous_v5h3_1199_200_006692861 - PH 1 TS 0534691 A - 16/01/1257 01:00 - H:M:S=0115:35:43 AVG= 0.78 DLT= 1.73
famous_v5h3_1199_200_006692861 - PH 1 TS 0535081 A - 01/02/1257 01:00 - H:M:S=0115:40:19 AVG= 0.78 DLT= 2.01
famous_v5h3_1199_200_006692861 - PH 1 TS 0535471 A - 16/02/1257 01:00 - H:M:S=0115:44:59 AVG= 0.78 DLT= 1.27
famous_v5h3_1199_200_006692861 - PH 1 TS 0535861 A - 01/03/1257 01:00 - H:M:S=0115:49:54 AVG= 0.78 DLT= 1.40
famous_v5h3_1199_200_006692861 - PH 1 TS 0536251 A - 16/03/1257 01:00 - H:M:S=0115:54:57 AVG= 0.78 DLT= 1.51
famous_v5h3_1199_200_006692861 - PH 1 TS 0536641 A - 01/04/1257 01:00 - H:M:S=0115:59:53 AVG= 0.78 DLT= 1.74
famous_v5h3_1199_200_006692861 - PH 1 TS 0537031 A - 16/04/1257 01:00 - H:M:S=0116:04:53 AVG= 0.78 DLT= 1.31
famous_v5h3_1199_200_006692861 - PH 1 TS 0537421 A - 01/05/1257 01:00 - H:M:S=0116:09:34 AVG= 0.78 DLT= 1.61

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 13444
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0537811 A - 16/05/1257 01:00 - H:M:S=0116:14:44 AVG= 0.78 DLT= 1.25
famous_v5h3_1199_200_006692861 - PH 1 TS 0538201 A - 01/06/1257 01:00 - H:M:S=0116:19:24 AVG= 0.78 DLT= 1.66
famous_v5h3_1199_200_006692861 - PH 1 TS 0538591 A - 16/06/1257 01:00 - H:M:S=0116:24:09 AVG= 0.78 DLT= 1.60
famous_v5h3_1199_200_006692861 - PH 1 TS 0538981 A - 01/07/1257 01:00 - H:M:S=0116:28:49 AVG= 0.78 DLT= 1.91
famous_v5h3_1199_200_006692861 - PH 1 TS 0539371 A - 16/07/1257 01:00 - H:M:S=0116:33:29 AVG= 0.78 DLT= 1.40
famous_v5h3_1199_200_006692861 - PH 1 TS 0539761 A - 01/08/1257 01:00 - H:M:S=0116:38:14 AVG= 0.78 DLT= 1.93
famous_v5h3_1199_200_006692861 - PH 1 TS 0540151 A - 16/08/1257 01:00 - H:M:S=0116:43:11 AVG= 0.78 DLT= 1.65
famous_v5h3_1199_200_006692861 - PH 1 TS 0540541 A - 01/09/1257 01:00 - H:M:S=0116:47:55 AVG= 0.78 DLT= 1.71
famous_v5h3_1199_200_006692861 - PH 1 TS 0540931 A - 16/09/1257 01:00 - H:M:S=0116:52:45 AVG= 0.78 DLT= 1.62
famous_v5h3_1199_200_006692861 - PH 1 TS 0541321 A - 01/10/1257 01:00 - H:M:S=0116:57:55 AVG= 0.78 DLT= 1.67
famous_v5h3_1199_200_006692861 - PH 1 TS 0541711 A - 16/10/1257 01:00 - H:M:S=0117:02:57 AVG= 0.78 DLT= 2.69
famous_v5h3_1199_200_006692861 - PH 1 TS 0542101 A - 01/11/1257 01:00 - H:M:S=0117:07:46 AVG= 0.78 DLT= 1.56
famous_v5h3_1199_200_006692861 - PH 1 TS 0542491 A - 16/11/1257 01:00 - H:M:S=0117:12:36 AVG= 0.78 DLT= 2.31
Trickling yearly means for 1257
Building file list for 1257 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1257.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0542881 A - 01/12/1257 01:00 - H:M:S=0117:17:37 AVG= 0.78 DLT= 1.62

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 15437
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_v5h3_1199_200_006692861 - PH 1 TS 0543271 A - 16/12/1257 01:00 - H:M:S=0117:24:40 AVG= 0.78 DLT= 1.57
famous_v5h3_1199_200_006692861 - PH 1 TS 0543661 A - 01/01/1258 01:00 - H:M:S=0117:29:28 AVG= 0.78 DLT= 1.27

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 15807
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0544051 A - 16/01/1258 01:00 - H:M:S=0117:38:21 AVG= 0.78 DLT= 1.71
famous_v5h3_1199_200_006692861 - PH 1 TS 0544441 A - 01/02/1258 01:00 - H:M:S=0117:43:15 AVG= 0.78 DLT= 1.43
famous_v5h3_1199_200_006692861 - PH 1 TS 0544831 A - 16/02/1258 01:00 - H:M:S=0117:48:03 AVG= 0.78 DLT= 1.48
famous_v5h3_1199_200_006692861 - PH 1 TS 0545221 A - 01/03/1258 01:00 - H:M:S=0117:52:50 AVG= 0.78 DLT= 1.74
famous_v5h3_1199_200_006692861 - PH 1 TS 0545611 A - 16/03/1258 01:00 - H:M:S=0117:57:42 AVG= 0.78 DLT= 1.76
famous_v5h3_1199_200_006692861 - PH 1 TS 0546001 A - 01/04/1258 01:00 - H:M:S=0118:02:35 AVG= 0.78 DLT= 1.30
famous_v5h3_1199_200_006692861 - PH 1 TS 0546391 A - 16/04/1258 01:00 - H:M:S=0118:07:27 AVG= 0.78 DLT= 1.26
famous_v5h3_1199_200_006692861 - PH 1 TS 0546781 A - 01/05/1258 01:00 - H:M:S=0118:12:17 AVG= 0.78 DLT= 1.20

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 17297
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 17446
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 17530
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0547171 A - 16/05/1258 01:00 - H:M:S=0118:27:34 AVG= 0.78 DLT= 2.07
famous_v5h3_1199_200_006692861 - PH 1 TS 0547561 A - 01/06/1258 01:00 - H:M:S=0118:32:14 AVG= 0.78 DLT= 1.78
famous_v5h3_1199_200_006692861 - PH 1 TS 0547951 A - 16/06/1258 01:00 - H:M:S=0118:37:04 AVG= 0.78 DLT= 1.99
famous_v5h3_1199_200_006692861 - PH 1 TS 0548341 A - 01/07/1258 01:00 - H:M:S=0118:41:52 AVG= 0.78 DLT= 1.83
famous_v5h3_1199_200_006692861 - PH 1 TS 0548731 A - 16/07/1258 01:00 - H:M:S=0118:46:44 AVG= 0.78 DLT= 2.12
famous_v5h3_1199_200_006692861 - PH 1 TS 0549121 A - 01/08/1258 01:00 - H:M:S=0118:51:30 AVG= 0.78 DLT= 1.24
famous_v5h3_1199_200_006692861 - PH 1 TS 0549511 A - 16/08/1258 01:00 - H:M:S=0118:56:19 AVG= 0.78 DLT= 1.49
famous_v5h3_1199_200_006692861 - PH 1 TS 0549901 A - 01/09/1258 01:00 - H:M:S=0119:01:06 AVG= 0.78 DLT= 1.78
famous_v5h3_1199_200_006692861 - PH 1 TS 0550291 A - 16/09/1258 01:00 - H:M:S=0119:05:59 AVG= 0.78 DLT= 1.15
famous_v5h3_1199_200_006692861 - PH 1 TS 0550681 A - 01/10/1258 01:00 - H:M:S=0119:10:48 AVG= 0.78 DLT= 1.75
famous_v5h3_1199_200_006692861 - PH 1 TS 0551071 A - 16/10/1258 01:00 - H:M:S=0119:15:40 AVG= 0.78 DLT= 1.60

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 19334
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0551461 A - 01/11/1258 01:00 - H:M:S=0119:24:54 AVG= 0.78 DLT= 1.81
famous_v5h3_1199_200_006692861 - PH 1 TS 0551851 A - 16/11/1258 01:00 - H:M:S=0119:29:41 AVG= 0.78 DLT= 2.11
Trickling yearly means for 1258
Building file list for 1258 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1258.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0552241 A - 01/12/1258 01:00 - H:M:S=0119:34:23 AVG= 0.78 DLT= 1.61

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 19754
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_v5h3_1199_200_006692861 - PH 1 TS 0552631 A - 16/12/1258 01:00 - H:M:S=0119:39:53 AVG= 0.78 DLT= 1.91
famous_v5h3_1199_200_006692861 - PH 1 TS 0553021 A - 01/01/1259 01:00 - H:M:S=0119:44:38 AVG= 0.78 DLT= 1.77
famous_v5h3_1199_200_006692861 - PH 1 TS 0553411 A - 16/01/1259 01:00 - H:M:S=0119:49:21 AVG= 0.78 DLT= 1.50

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 20320
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0553801 A - 01/02/1259 01:00 - H:M:S=0119:57:39 AVG= 0.78 DLT= 1.90
famous_v5h3_1199_200_006692861 - PH 1 TS 0554191 A - 16/02/1259 01:00 - H:M:S=0120:02:37 AVG= 0.78 DLT= 1.49
famous_v5h3_1199_200_006692861 - PH 1 TS 0554581 A - 01/03/1259 01:00 - H:M:S=0120:07:25 AVG= 0.78 DLT= 1.20
famous_v5h3_1199_200_006692861 - PH 1 TS 0554971 A - 16/03/1259 01:00 - H:M:S=0120:12:09 AVG= 0.78 DLT= 1.70
famous_v5h3_1199_200_006692861 - PH 1 TS 0555361 A - 01/04/1259 01:00 - H:M:S=0120:16:58 AVG= 0.78 DLT= 1.44
famous_v5h3_1199_200_006692861 - PH 1 TS 0555751 A - 16/04/1259 01:00 - H:M:S=0120:21:46 AVG= 0.78 DLT= 1.53
famous_v5h3_1199_200_006692861 - PH 1 TS 0556141 A - 01/05/1259 01:00 - H:M:S=0120:26:31 AVG= 0.78 DLT= 1.66
famous_v5h3_1199_200_006692861 - PH 1 TS 0556531 A - 16/05/1259 01:00 - H:M:S=0120:31:25 AVG= 0.78 DLT= 1.30
famous_v5h3_1199_200_006692861 - PH 1 TS 0556921 A - 01/06/1259 01:00 - H:M:S=0120:36:17 AVG= 0.78 DLT= 1.25
famous_v5h3_1199_200_006692861 - PH 1 TS 0557311 A - 16/06/1259 01:00 - H:M:S=0120:41:02 AVG= 0.78 DLT= 1.86
famous_v5h3_1199_200_006692861 - PH 1 TS 0557701 A - 01/07/1259 01:00 - H:M:S=0120:45:49 AVG= 0.78 DLT= 1.89
famous_v5h3_1199_200_006692861 - PH 1 TS 0558091 A - 16/07/1259 01:00 - H:M:S=0120:50:36 AVG= 0.78 DLT= 1.24
famous_v5h3_1199_200_006692861 - PH 1 TS 0558481 A - 01/08/1259 01:00 - H:M:S=0120:55:21 AVG= 0.78 DLT= 1.84
famous_v5h3_1199_200_006692861 - PH 1 TS 0558871 A - 16/08/1259 01:00 - H:M:S=0121:00:03 AVG= 0.78 DLT= 1.23
famous_v5h3_1199_200_006692861 - PH 1 TS 0559261 A - 01/09/1259 01:00 - H:M:S=0121:04:52 AVG= 0.78 DLT= 1.85
famous_v5h3_1199_200_006692861 - PH 1 TS 0559651 A - 16/09/1259 01:00 - H:M:S=0121:09:46 AVG= 0.78 DLT= 2.13
famous_v5h3_1199_200_006692861 - PH 1 TS 0560041 A - 01/10/1259 01:00 - H:M:S=0121:14:34 AVG= 0.78 DLT= 1.28
famous_v5h3_1199_200_006692861 - PH 1 TS 0560431 A - 16/10/1259 01:00 - H:M:S=0121:19:35 AVG= 0.78 DLT= 1.28

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 22999
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 23047
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0560821 A - 01/11/1259 01:00 - H:M:S=0121:31:47 AVG= 0.78 DLT= 1.76
famous_v5h3_1199_200_006692861 - PH 1 TS 0561211 A - 16/11/1259 01:00 - H:M:S=0121:36:33 AVG= 0.78 DLT= 1.56
Trickling yearly means for 1259
Building file list for 1259 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1259.zip for upload...
Processing decadal means for 1250 through 1259
Queuing intermediate upload for CPDN/BOINC: cpdnout6.zip
famous_v5h3_1199_200_006692861 - PH 1 TS 0561991 A - 16/12/1259 01:00 - H:M:S=0121:46:22 AVG= 0.78 DLT= 1.40
famous_v5h3_1199_200_006692861 - PH 1 TS 0562381 A - 01/01/1260 01:00 - H:M:S=0121:51:18 AVG= 0.78 DLT= 1.64

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 23501
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 23544
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0562771 A - 16/01/1260 01:00 - H:M:S=0122:05:13 AVG= 0.78 DLT= 1.68
famous_v5h3_1199_200_006692861 - PH 1 TS 0563161 A - 01/02/1260 01:00 - H:M:S=0122:10:19 AVG= 0.78 DLT= 1.39

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 23765
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0563551 A - 16/02/1260 01:00 - H:M:S=0122:17:42 AVG= 0.78 DLT= 2.01
famous_v5h3_1199_200_006692861 - PH 1 TS 0563941 A - 01/03/1260 01:00 - H:M:S=0122:22:27 AVG= 0.78 DLT= 2.25
famous_v5h3_1199_200_006692861 - PH 1 TS 0564331 A - 16/03/1260 01:00 - H:M:S=0122:27:11 AVG= 0.78 DLT= 1.37
famous_v5h3_1199_200_006692861 - PH 1 TS 0564721 A - 01/04/1260 01:00 - H:M:S=0122:31:58 AVG= 0.78 DLT= 2.04
famous_v5h3_1199_200_006692861 - PH 1 TS 0565111 A - 16/04/1260 01:00 - H:M:S=0122:36:39 AVG= 0.78 DLT= 1.55
famous_v5h3_1199_200_006692861 - PH 1 TS 0565501 A - 01/05/1260 01:00 - H:M:S=0122:41:29 AVG= 0.78 DLT= 1.60
famous_v5h3_1199_200_006692861 - PH 1 TS 0565891 A - 16/05/1260 01:00 - H:M:S=0122:46:10 AVG= 0.78 DLT= 1.42
famous_v5h3_1199_200_006692861 - PH 1 TS 0566281 A - 01/06/1260 01:00 - H:M:S=0122:51:01 AVG= 0.78 DLT= 1.48

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 24451
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 24467
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0566671 A - 16/06/1260 01:00 - H:M:S=0123:01:13 AVG= 0.78 DLT= 1.10
famous_v5h3_1199_200_006692861 - PH 1 TS 0567061 A - 01/07/1260 01:00 - H:M:S=0123:05:53 AVG= 0.78 DLT= 1.94
famous_v5h3_1199_200_006692861 - PH 1 TS 0567451 A - 16/07/1260 01:00 - H:M:S=0123:10:38 AVG= 0.78 DLT= 1.53
famous_v5h3_1199_200_006692861 - PH 1 TS 0567841 A - 01/08/1260 01:00 - H:M:S=0123:15:24 AVG= 0.78 DLT= 1.67
famous_v5h3_1199_200_006692861 - PH 1 TS 0568231 A - 16/08/1260 01:00 - H:M:S=0123:20:09 AVG= 0.78 DLT= 1.80
famous_v5h3_1199_200_006692861 - PH 1 TS 0568621 A - 01/09/1260 01:00 - H:M:S=0123:24:58 AVG= 0.78 DLT= 1.20
famous_v5h3_1199_200_006692861 - PH 1 TS 0569011 A - 16/09/1260 01:00 - H:M:S=0123:29:46 AVG= 0.78 DLT= 1.19
famous_v5h3_1199_200_006692861 - PH 1 TS 0569401 A - 01/10/1260 01:00 - H:M:S=0123:34:35 AVG= 0.78 DLT= 1.11
famous_v5h3_1199_200_006692861 - PH 1 TS 0569791 A - 16/10/1260 01:00 - H:M:S=0123:39:26 AVG= 0.78 DLT= 1.26
famous_v5h3_1199_200_006692861 - PH 1 TS 0570181 A - 01/11/1260 01:00 - H:M:S=0123:44:20 AVG= 0.78 DLT= 1.32
famous_v5h3_1199_200_006692861 - PH 1 TS 0570571 A - 16/11/1260 01:00 - H:M:S=0123:49:06 AVG= 0.78 DLT= 1.41
Trickling yearly means for 1260
Building file list for 1260 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1260.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0570961 A - 01/12/1260 01:00 - H:M:S=0123:53:53 AVG= 0.78 DLT= 1.56
famous_v5h3_1199_200_006692861 - PH 1 TS 0571351 A - 16/12/1260 01:00 - H:M:S=0123:58:42 AVG= 0.78 DLT= 1.75
famous_v5h3_1199_200_006692861 - PH 1 TS 0571741 A - 01/01/1261 01:00 - H:M:S=0124:03:23 AVG= 0.78 DLT= 1.84
famous_v5h3_1199_200_006692861 - PH 1 TS 0572131 A - 16/01/1261 01:00 - H:M:S=0124:08:05 AVG= 0.78 DLT= 1.76
famous_v5h3_1199_200_006692861 - PH 1 TS 0572521 A - 01/02/1261 01:00 - H:M:S=0124:12:47 AVG= 0.78 DLT= 1.62
famous_v5h3_1199_200_006692861 - PH 1 TS 0572911 A - 16/02/1261 01:00 - H:M:S=0124:17:31 AVG= 0.78 DLT= 1.75
famous_v5h3_1199_200_006692861 - PH 1 TS 0573301 A - 01/03/1261 01:00 - H:M:S=0124:22:19 AVG= 0.78 DLT= 1.59
famous_v5h3_1199_200_006692861 - PH 1 TS 0573691 A - 16/03/1261 01:00 - H:M:S=0124:27:08 AVG= 0.78 DLT= 2.00
famous_v5h3_1199_200_006692861 - PH 1 TS 0574081 A - 01/04/1261 01:00 - H:M:S=0124:32:00 AVG= 0.78 DLT= 2.02
famous_v5h3_1199_200_006692861 - PH 1 TS 0574471 A - 16/04/1261 01:00 - H:M:S=0124:36:45 AVG= 0.78 DLT= 1.96
famous_v5h3_1199_200_006692861 - PH 1 TS 0574861 A - 01/05/1261 01:00 - H:M:S=0124:41:42 AVG= 0.78 DLT= 1.26
famous_v5h3_1199_200_006692861 - PH 1 TS 0575251 A - 16/05/1261 01:00 - H:M:S=0124:46:27 AVG= 0.78 DLT= 1.66
famous_v5h3_1199_200_006692861 - PH 1 TS 0575641 A - 01/06/1261 01:00 - H:M:S=0124:51:16 AVG= 0.78 DLT= 2.11

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 27246
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0576031 A - 16/06/1261 01:00 - H:M:S=0125:00:30 AVG= 0.78 DLT= 1.95
famous_v5h3_1199_200_006692861 - PH 1 TS 0576421 A - 01/07/1261 01:00 - H:M:S=0125:05:20 AVG= 0.78 DLT= 2.01
famous_v5h3_1199_200_006692861 - PH 1 TS 0576811 A - 16/07/1261 01:00 - H:M:S=0125:10:17 AVG= 0.78 DLT= 1.88
famous_v5h3_1199_200_006692861 - PH 1 TS 0577201 A - 01/08/1261 01:00 - H:M:S=0125:15:05 AVG= 0.78 DLT= 1.73
famous_v5h3_1199_200_006692861 - PH 1 TS 0577591 A - 16/08/1261 01:00 - H:M:S=0125:19:48 AVG= 0.78 DLT= 1.43
famous_v5h3_1199_200_006692861 - PH 1 TS 0577981 A - 01/09/1261 01:00 - H:M:S=0125:24:41 AVG= 0.78 DLT= 1.53
famous_v5h3_1199_200_006692861 - PH 1 TS 0578371 A - 16/09/1261 01:00 - H:M:S=0125:29:31 AVG= 0.78 DLT= 2.21

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 28432
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0578761 A - 01/10/1261 01:00 - H:M:S=0125:39:09 AVG= 0.78 DLT= 1.99
famous_v5h3_1199_200_006692861 - PH 1 TS 0579151 A - 16/10/1261 01:00 - H:M:S=0125:44:04 AVG= 0.78 DLT= 1.82
famous_v5h3_1199_200_006692861 - PH 1 TS 0579541 A - 01/11/1261 01:00 - H:M:S=0125:48:50 AVG= 0.78 DLT= 1.19
famous_v5h3_1199_200_006692861 - PH 1 TS 0579931 A - 16/11/1261 01:00 - H:M:S=0125:53:36 AVG= 0.78 DLT= 1.42
Trickling yearly means for 1261
Building file list for 1261 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1261.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0580321 A - 01/12/1261 01:00 - H:M:S=0125:58:14 AVG= 0.78 DLT= 1.33
famous_v5h3_1199_200_006692861 - PH 1 TS 0580711 A - 16/12/1261 01:00 - H:M:S=0126:03:01 AVG= 0.78 DLT= 1.29

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 29338
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0581101 A - 01/01/1262 01:00 - H:M:S=0126:09:01 AVG= 0.78 DLT= 1.27

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 29543
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0581491 A - 16/01/1262 01:00 - H:M:S=0126:16:05 AVG= 0.78 DLT= 1.23
famous_v5h3_1199_200_006692861 - PH 1 TS 0581881 A - 01/02/1262 01:00 - H:M:S=0126:20:48 AVG= 0.78 DLT= 1.34

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 29960
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 29982
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0582271 A - 16/02/1262 01:00 - H:M:S=0126:31:19 AVG= 0.78 DLT= 1.96
famous_v5h3_1199_200_006692861 - PH 1 TS 0582661 A - 01/03/1262 01:00 - H:M:S=0126:36:00 AVG= 0.78 DLT= 1.89
famous_v5h3_1199_200_006692861 - PH 1 TS 0583051 A - 16/03/1262 01:00 - H:M:S=0126:40:41 AVG= 0.78 DLT= 1.76
famous_v5h3_1199_200_006692861 - PH 1 TS 0583441 A - 01/04/1262 01:00 - H:M:S=0126:45:25 AVG= 0.78 DLT= 1.77
famous_v5h3_1199_200_006692861 - PH 1 TS 0583831 A - 16/04/1262 01:00 - H:M:S=0126:50:16 AVG= 0.78 DLT= 1.53
famous_v5h3_1199_200_006692861 - PH 1 TS 0584221 A - 01/05/1262 01:00 - H:M:S=0126:54:58 AVG= 0.78 DLT= 1.27
famous_v5h3_1199_200_006692861 - PH 1 TS 0584611 A - 16/05/1262 01:00 - H:M:S=0126:59:46 AVG= 0.78 DLT= 1.74

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 31201
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0585001 A - 01/06/1262 01:00 - H:M:S=0127:09:13 AVG= 0.78 DLT= 1.80
famous_v5h3_1199_200_006692861 - PH 1 TS 0585391 A - 16/06/1262 01:00 - H:M:S=0127:13:57 AVG= 0.78 DLT= 2.06
famous_v5h3_1199_200_006692861 - PH 1 TS 0585781 A - 01/07/1262 01:00 - H:M:S=0127:18:43 AVG= 0.78 DLT= 1.99
famous_v5h3_1199_200_006692861 - PH 1 TS 0586171 A - 16/07/1262 01:00 - H:M:S=0127:23:33 AVG= 0.78 DLT= 1.19
famous_v5h3_1199_200_006692861 - PH 1 TS 0586561 A - 01/08/1262 01:00 - H:M:S=0127:28:27 AVG= 0.78 DLT= 1.23
famous_v5h3_1199_200_006692861 - PH 1 TS 0586951 A - 16/08/1262 01:00 - H:M:S=0127:33:12 AVG= 0.78 DLT= 1.62
famous_v5h3_1199_200_006692861 - PH 1 TS 0587341 A - 01/09/1262 01:00 - H:M:S=0127:38:04 AVG= 0.78 DLT= 1.80
famous_v5h3_1199_200_006692861 - PH 1 TS 0587731 A - 16/09/1262 01:00 - H:M:S=0127:43:03 AVG= 0.78 DLT= 1.82
famous_v5h3_1199_200_006692861 - PH 1 TS 0588121 A - 01/10/1262 01:00 - H:M:S=0127:47:50 AVG= 0.78 DLT= 1.34
famous_v5h3_1199_200_006692861 - PH 1 TS 0588511 A - 16/10/1262 01:00 - H:M:S=0127:52:37 AVG= 0.78 DLT= 1.28
famous_v5h3_1199_200_006692861 - PH 1 TS 0588901 A - 01/11/1262 01:00 - H:M:S=0127:57:19 AVG= 0.78 DLT= 1.42
famous_v5h3_1199_200_006692861 - PH 1 TS 0589291 A - 16/11/1262 01:00 - H:M:S=0128:02:08 AVG= 0.78 DLT= 1.70

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 541
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Trickling yearly means for 1262
Building file list for 1262 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1262.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0589681 A - 01/12/1262 01:00 - H:M:S=0128:08:54 AVG= 0.78 DLT= 2.13
famous_v5h3_1199_200_006692861 - PH 1 TS 0590071 A - 16/12/1262 01:00 - H:M:S=0128:13:38 AVG= 0.78 DLT= 2.08
famous_v5h3_1199_200_006692861 - PH 1 TS 0590461 A - 01/01/1263 01:00 - H:M:S=0128:18:18 AVG= 0.78 DLT= 1.74
famous_v5h3_1199_200_006692861 - PH 1 TS 0590851 A - 16/01/1263 01:00 - H:M:S=0128:22:59 AVG= 0.78 DLT= 1.86
famous_v5h3_1199_200_006692861 - PH 1 TS 0591241 A - 01/02/1263 01:00 - H:M:S=0128:27:41 AVG= 0.78 DLT= 1.66
famous_v5h3_1199_200_006692861 - PH 1 TS 0591631 A - 16/02/1263 01:00 - H:M:S=0128:32:32 AVG= 0.78 DLT= 1.10
famous_v5h3_1199_200_006692861 - PH 1 TS 0592021 A - 01/03/1263 01:00 - H:M:S=0128:37:21 AVG= 0.78 DLT= 1.55
famous_v5h3_1199_200_006692861 - PH 1 TS 0592411 A - 16/03/1263 01:00 - H:M:S=0128:42:04 AVG= 0.78 DLT= 1.75
famous_v5h3_1199_200_006692861 - PH 1 TS 0592801 A - 01/04/1263 01:00 - H:M:S=0128:46:52 AVG= 0.78 DLT= 1.65

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 2060
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0593191 A - 16/04/1263 01:00 - H:M:S=0128:56:29 AVG= 0.78 DLT= 1.64
famous_v5h3_1199_200_006692861 - PH 1 TS 0593581 A - 01/05/1263 01:00 - H:M:S=0129:01:22 AVG= 0.78 DLT= 1.67

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 2480
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0593971 A - 16/05/1263 01:00 - H:M:S=0129:10:13 AVG= 0.78 DLT= 1.22
famous_v5h3_1199_200_006692861 - PH 1 TS 0594361 A - 01/06/1263 01:00 - H:M:S=0129:14:58 AVG= 0.78 DLT= 1.40
famous_v5h3_1199_200_006692861 - PH 1 TS 0594751 A - 16/06/1263 01:00 - H:M:S=0129:19:37 AVG= 0.78 DLT= 1.28
famous_v5h3_1199_200_006692861 - PH 1 TS 0595141 A - 01/07/1263 01:00 - H:M:S=0129:24:28 AVG= 0.78 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 3070
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0595141 A - 01/07/1263 01:00 - H:M:S=0129:24:30 AVG= 0.78 DLT= 0.00
famous_v5h3_1199_200_006692861 - PH 1 TS 0595531 A - 16/07/1263 01:00 - H:M:S=0129:29:17 AVG= 0.78 DLT= 1.82
famous_v5h3_1199_200_006692861 - PH 1 TS 0595921 A - 01/08/1263 01:00 - H:M:S=0129:33:59 AVG= 0.78 DLT= 1.30
famous_v5h3_1199_200_006692861 - PH 1 TS 0596311 A - 16/08/1263 01:00 - H:M:S=0129:38:57 AVG= 0.78 DLT= 1.11
famous_v5h3_1199_200_006692861 - PH 1 TS 0596701 A - 01/09/1263 01:00 - H:M:S=0129:43:58 AVG= 0.78 DLT= 1.59
famous_v5h3_1199_200_006692861 - PH 1 TS 0597091 A - 16/09/1263 01:00 - H:M:S=0129:48:47 AVG= 0.78 DLT= 2.02
famous_v5h3_1199_200_006692861 - PH 1 TS 0597481 A - 01/10/1263 01:00 - H:M:S=0129:53:35 AVG= 0.78 DLT= 1.71
famous_v5h3_1199_200_006692861 - PH 1 TS 0597871 A - 16/10/1263 01:00 - H:M:S=0129:58:21 AVG= 0.78 DLT= 2.00
famous_v5h3_1199_200_006692861 - PH 1 TS 0598261 A - 01/11/1263 01:00 - H:M:S=0130:03:08 AVG= 0.78 DLT= 1.16
famous_v5h3_1199_200_006692861 - PH 1 TS 0598651 A - 16/11/1263 01:00 - H:M:S=0130:08:01 AVG= 0.78 DLT= 1.36
Trickling yearly means for 1263
Building file list for 1263 in dataout -- 4 entries
Creating trickle file trickle_famous_v5h3_1199_200_006692861_0_1263.zip for upload...
famous_v5h3_1199_200_006692861 - PH 1 TS 0599041 A - 01/12/1263 01:00 - H:M:S=0130:12:49 AVG= 0.78 DLT= 1.78
famous_v5h3_1199_200_006692861 - PH 1 TS 0599431 A - 16/12/1263 01:00 - H:M:S=0130:17:28 AVG= 0.78 DLT= 1.42
famous_v5h3_1199_200_006692861 - PH 1 TS 0599821 A - 01/01/1264 01:00 - H:M:S=0130:22:11 AVG= 0.78 DLT= 1.96
famous_v5h3_1199_200_006692861 - PH 1 TS 0600211 A - 16/01/1264 01:00 - H:M:S=0130:26:56 AVG= 0.78 DLT= 1.37
famous_v5h3_1199_200_006692861 - PH 1 TS 0600601 A - 01/02/1264 01:00 - H:M:S=0130:31:40 AVG= 0.78 DLT= 1.39
famous_v5h3_1199_200_006692861 - PH 1 TS 0600991 A - 16/02/1264 01:00 - H:M:S=0130:36:31 AVG= 0.78 DLT= 1.48

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 5385
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Program launched with process id # 5391
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
famous_v5h3_1199_200_006692861 - PH 1 TS 0601381 A - 01/03/1264 01:00 - H:M:S=0130:44:22 AVG= 0.78 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 5534
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0601381 A - 01/03/1264 01:00 - H:M:S=0130:44:26 AVG= 0.78 DLT= 0.00
famous_v5h3_1199_200_006692861 - PH 1 TS 0601771 A - 16/03/1264 01:00 - H:M:S=0130:49:10 AVG= 0.78 DLT= 1.44
famous_v5h3_1199_200_006692861 - PH 1 TS 0602161 A - 01/04/1264 01:00 - H:M:S=0130:54:01 AVG= 0.78 DLT= 1.36
famous_v5h3_1199_200_006692861 - PH 1 TS 0602551 A - 16/04/1264 01:00 - H:M:S=0130:58:42 AVG= 0.78 DLT= 1.36
famous_v5h3_1199_200_006692861 - PH 1 TS 0602941 A - 01/05/1264 01:00 - H:M:S=0131:03:30 AVG= 0.78 DLT= 1.23
famous_v5h3_1199_200_006692861 - PH 1 TS 0603331 A - 16/05/1264 01:00 - H:M:S=0131:08:16 AVG= 0.78 DLT= 1.40
famous_v5h3_1199_200_006692861 - PH 1 TS 0603721 A - 01/06/1264 01:00 - H:M:S=0131:13:15 AVG= 0.78 DLT= 1.50
famous_v5h3_1199_200_006692861 - PH 1 TS 0604111 A - 16/06/1264 01:00 - H:M:S=0131:18:11 AVG= 0.78 DLT= 1.95
famous_v5h3_1199_200_006692861 - PH 1 TS 0604501 A - 01/07/1264 01:00 - H:M:S=0131:23:24 AVG= 0.78 DLT= 1.54
famous_v5h3_1199_200_006692861 - PH 1 TS 0604891 A - 16/07/1264 01:00 - H:M:S=0131:28:48 AVG= 0.78 DLT= 1.75
famous_v5h3_1199_200_006692861 - PH 1 TS 0605281 A - 01/08/1264 01:00 - H:M:S=0131:34:18 AVG= 0.78 DLT= 1.51
famous_v5h3_1199_200_006692861 - PH 1 TS 0605671 A - 16/08/1264 01:00 - H:M:S=0131:39:50 AVG= 0.78 DLT= 1.69

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 8257
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0606061 A - 01/09/1264 01:00 - H:M:S=0131:48:05 AVG= 0.78 DLT= 1.80
famous_v5h3_1199_200_006692861 - PH 1 TS 0606451 A - 16/09/1264 01:00 - H:M:S=0131:53:40 AVG= 0.78 DLT= 1.48
famous_v5h3_1199_200_006692861 - PH 1 TS 0606841 A - 01/10/1264 01:00 - H:M:S=0131:59:16 AVG= 0.78 DLT= 2.00

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 9795
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 9851
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0607231 A - 16/10/1264 01:00 - H:M:S=0132:11:18 AVG= 0.78 DLT= 1.83

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 10455
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 10634
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 10641
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 10880
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v5h3_1199_200_006692861 - PH 1 TS 0607621 A - 01/11/1264 01:00 - H:M:S=0132:22:59 AVG= 0.78 DLT= 1.66

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 11460
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 11630
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 11713
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 11934
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 5...
Starting model ID famous_v5h3_1199_200_006692861   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 133490 /home/jsc/BOINC/slots/0
Program launched with process id # 12094
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
Model restart required...
Sorry, too many model crashes! :-(
Cleaning up from the run...
 (6358): called boinc_finish
Detaching shared memory...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
21 Sep 2010 15:59:34 922557 11724074 famous_v5h3_1199_200_006692861_0 599,066 468,769 0.7825
21 Sep 2010 13:16:43 922557 11724074 famous_v5h3_1199_200_006692861_0 589,706 461,334 0.7823
21 Sep 2010 10:24:49 922557 11724074 famous_v5h3_1199_200_006692861_0 580,346 453,494 0.7814
21 Sep 2010 07:40:56 922557 11724074 famous_v5h3_1199_200_006692861_0 570,986 446,033 0.7812
21 Sep 2010 04:42:27 922557 11724074 famous_v5h3_1199_200_006692861_0 561,626 438,080 0.7800
21 Sep 2010 02:56:27 922557 11724074 famous_v5h3_1199_200_006692861_0 552,266 430,463 0.7794
20 Sep 2010 22:54:39 922557 11724074 famous_v5h3_1199_200_006692861_0 542,906 422,257 0.7778
20 Sep 2010 20:07:16 922557 11724074 famous_v5h3_1199_200_006692861_0 533,546 415,071 0.7779
20 Sep 2010 17:04:30 922557 11724074 famous_v5h3_1199_200_006692861_0 524,186 407,439 0.7773
20 Sep 2010 13:59:31 922557 11724074 famous_v5h3_1199_200_006692861_0 514,826 399,212 0.7754
20 Sep 2010 10:43:17 922557 11724074 famous_v5h3_1199_200_006692861_0 505,466 390,414 0.7724
20 Sep 2010 08:19:56 922557 11724074 famous_v5h3_1199_200_006692861_0 496,106 382,654 0.7713
20 Sep 2010 04:41:48 922557 11724074 famous_v5h3_1199_200_006692861_0 486,746 374,546 0.7695
20 Sep 2010 03:06:32 922557 11724074 famous_v5h3_1199_200_006692861_0 477,386 367,164 0.7691
19 Sep 2010 23:29:11 922557 11724074 famous_v5h3_1199_200_006692861_0 468,026 360,056 0.7693
19 Sep 2010 20:56:19 922557 11724074 famous_v5h3_1199_200_006692861_0 458,666 352,903 0.7694
19 Sep 2010 17:42:25 922557 11724074 famous_v5h3_1199_200_006692861_0 449,306 344,502 0.7667
19 Sep 2010 14:00:54 922557 11724074 famous_v5h3_1199_200_006692861_0 439,946 337,232 0.7665
19 Sep 2010 08:24:17 922557 11724074 famous_v5h3_1199_200_006692861_0 430,586 330,073 0.7666
19 Sep 2010 03:04:39 922557 11724074 famous_v5h3_1199_200_006692861_0 421,226 322,384 0.7653


©2024 climateprediction.net