climateprediction.net home page
Task 12174390

Task 12174390

Name famous_w0oo_599_200_006747576_2
Workunit 6950892
Created 20 Nov 2010, 13:20:59 UTC
Sent 20 Nov 2010, 13:57:35 UTC
Report deadline 19 Feb 2011, 21:24:46 UTC
Received 27 Nov 2010, 22:41:41 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 922557
Run time
CPU time 3 days 16 hours 29 min 38 sec
Validate state Invalid
Credit 1,235.35
Device peak FLOPS 1.64 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>5.10.45</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 12389
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0339691 A - 16/03/0636 01:00 - H:M:S=0078:07:46 AVG= 0.83 DLT= 1.44

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 12841
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 13125
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0340081 A - 01/04/0636 01:00 - H:M:S=0078:18:22 AVG= 0.83 DLT= 1.42
famous_w0oo_599_200_006747576 - PH 1 TS 0340471 A - 16/04/0636 01:00 - H:M:S=0078:23:13 AVG= 0.83 DLT= 1.29

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 13783
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0340861 A - 01/05/0636 01:00 - H:M:S=0078:28:33 AVG= 0.83 DLT= 1.40
famous_w0oo_599_200_006747576 - PH 1 TS 0341251 A - 16/05/0636 01:00 - H:M:S=0078:33:30 AVG= 0.83 DLT= 1.63
famous_w0oo_599_200_006747576 - PH 1 TS 0341641 A - 01/06/0636 01:00 - H:M:S=0078:38:28 AVG= 0.83 DLT= 1.48
famous_w0oo_599_200_006747576 - PH 1 TS 0342031 A - 16/06/0636 01:00 - H:M:S=0078:43:26 AVG= 0.83 DLT= 1.57
famous_w0oo_599_200_006747576 - PH 1 TS 0342421 A - 01/07/0636 01:00 - H:M:S=0078:48:20 AVG= 0.83 DLT= 2.02
famous_w0oo_599_200_006747576 - PH 1 TS 0342811 A - 16/07/0636 01:00 - H:M:S=0078:53:14 AVG= 0.83 DLT= 1.57
famous_w0oo_599_200_006747576 - PH 1 TS 0343201 A - 01/08/0636 01:00 - H:M:S=0078:58:12 AVG= 0.83 DLT= 1.49
famous_w0oo_599_200_006747576 - PH 1 TS 0343591 A - 16/08/0636 01:00 - H:M:S=0079:03:07 AVG= 0.83 DLT= 1.33
famous_w0oo_599_200_006747576 - PH 1 TS 0343981 A - 01/09/0636 01:00 - H:M:S=0079:08:08 AVG= 0.83 DLT= 1.90
famous_w0oo_599_200_006747576 - PH 1 TS 0344371 A - 16/09/0636 01:00 - H:M:S=0079:13:07 AVG= 0.83 DLT= 1.48
famous_w0oo_599_200_006747576 - PH 1 TS 0344761 A - 01/10/0636 01:00 - H:M:S=0079:18:09 AVG= 0.83 DLT= 1.35
famous_w0oo_599_200_006747576 - PH 1 TS 0345151 A - 16/10/0636 01:00 - H:M:S=0079:23:08 AVG= 0.83 DLT= 1.25

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 17774
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0345541 A - 01/11/0636 01:00 - H:M:S=0079:30:14 AVG= 0.83 DLT= 1.45
famous_w0oo_599_200_006747576 - PH 1 TS 0345931 A - 16/11/0636 01:00 - H:M:S=0079:35:13 AVG= 0.83 DLT= 1.61

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 18570
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 18629
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Trickling yearly means for 636
Building file list for 636 in dataout -- 4 entries
Creating trickle file trickle_famous_w0oo_599_200_006747576_2_636.zip for upload...
famous_w0oo_599_200_006747576 - PH 1 TS 0346321 A - 01/12/0636 01:00 - H:M:S=0079:42:55 AVG= 0.83 DLT= 1.60
famous_w0oo_599_200_006747576 - PH 1 TS 0346711 A - 16/12/0636 01:00 - H:M:S=0079:47:49 AVG= 0.83 DLT= 1.45
famous_w0oo_599_200_006747576 - PH 1 TS 0347101 A - 01/01/0637 01:00 - H:M:S=0079:52:40 AVG= 0.83 DLT= 1.31
famous_w0oo_599_200_006747576 - PH 1 TS 0347491 A - 16/01/0637 01:00 - H:M:S=0079:57:32 AVG= 0.83 DLT= 1.34
famous_w0oo_599_200_006747576 - PH 1 TS 0347881 A - 01/02/0637 01:00 - H:M:S=0080:02:26 AVG= 0.83 DLT= 1.34
famous_w0oo_599_200_006747576 - PH 1 TS 0348271 A - 16/02/0637 01:00 - H:M:S=0080:07:20 AVG= 0.83 DLT= 1.50
famous_w0oo_599_200_006747576 - PH 1 TS 0348661 A - 01/03/0637 01:00 - H:M:S=0080:12:20 AVG= 0.83 DLT= 1.46

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 21105
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 21186
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0349051 A - 16/03/0637 01:00 - H:M:S=0080:20:56 AVG= 0.83 DLT= 1.34

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 21602
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0349441 A - 01/04/0637 01:00 - H:M:S=0080:27:11 AVG= 0.83 DLT= 1.76
famous_w0oo_599_200_006747576 - PH 1 TS 0349831 A - 16/04/0637 01:00 - H:M:S=0080:32:14 AVG= 0.83 DLT= 1.46

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 22577
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0350221 A - 01/05/0637 01:00 - H:M:S=0080:41:26 AVG= 0.83 DLT= 1.75
famous_w0oo_599_200_006747576 - PH 1 TS 0350611 A - 16/05/0637 01:00 - H:M:S=0080:46:24 AVG= 0.83 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 23433
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 23613
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0351001 A - 01/06/0637 01:00 - H:M:S=0080:57:02 AVG= 0.83 DLT= 1.52
famous_w0oo_599_200_006747576 - PH 1 TS 0351391 A - 16/06/0637 01:00 - H:M:S=0081:01:49 AVG= 0.83 DLT= 1.48
famous_w0oo_599_200_006747576 - PH 1 TS 0351781 A - 01/07/0637 01:00 - H:M:S=0081:06:49 AVG= 0.83 DLT= 1.73
famous_w0oo_599_200_006747576 - PH 1 TS 0352171 A - 16/07/0637 01:00 - H:M:S=0081:11:47 AVG= 0.83 DLT= 1.30

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Program launched with process id # 24977
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
famous_w0oo_599_200_006747576 - PH 1 TS 0352171 A - 16/07/0637 01:00 - H:M:S=0081:12:04 AVG= 0.83 DLT= 0.00
famous_w0oo_599_200_006747576 - PH 1 TS 0352561 A - 01/08/0637 01:00 - H:M:S=0081:17:08 AVG= 0.83 DLT= 1.19

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 25614
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0352951 A - 16/08/0637 01:00 - H:M:S=0081:26:08 AVG= 0.83 DLT= 1.52
famous_w0oo_599_200_006747576 - PH 1 TS 0353341 A - 01/09/0637 01:00 - H:M:S=0081:31:14 AVG= 0.83 DLT= 1.27
famous_w0oo_599_200_006747576 - PH 1 TS 0353731 A - 16/09/0637 01:00 - H:M:S=0081:36:16 AVG= 0.83 DLT= 1.82
famous_w0oo_599_200_006747576 - PH 1 TS 0354121 A - 01/10/0637 01:00 - H:M:S=0081:41:23 AVG= 0.83 DLT= 1.84

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 27098
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0354511 A - 16/10/0637 01:00 - H:M:S=0081:50:01 AVG= 0.83 DLT= 1.47
famous_w0oo_599_200_006747576 - PH 1 TS 0354901 A - 01/11/0637 01:00 - H:M:S=0081:55:01 AVG= 0.83 DLT= 1.24

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 27745
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0355291 A - 16/11/0637 01:00 - H:M:S=0082:00:22 AVG= 0.83 DLT= 1.59
Trickling yearly means for 637
Building file list for 637 in dataout -- 4 entries
Creating trickle file trickle_famous_w0oo_599_200_006747576_2_637.zip for upload...
famous_w0oo_599_200_006747576 - PH 1 TS 0355681 A - 01/12/0637 01:00 - H:M:S=0082:05:15 AVG= 0.83 DLT= 2.20
famous_w0oo_599_200_006747576 - PH 1 TS 0356071 A - 16/12/0637 01:00 - H:M:S=0082:10:07 AVG= 0.83 DLT= 1.18
famous_w0oo_599_200_006747576 - PH 1 TS 0356461 A - 01/01/0638 01:00 - H:M:S=0082:15:03 AVG= 0.83 DLT= 1.38

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 29310
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0356851 A - 16/01/0638 01:00 - H:M:S=0082:22:06 AVG= 0.83 DLT= 1.46
famous_w0oo_599_200_006747576 - PH 1 TS 0357241 A - 01/02/0638 01:00 - H:M:S=0082:27:01 AVG= 0.83 DLT= 1.87
famous_w0oo_599_200_006747576 - PH 1 TS 0357631 A - 16/02/0638 01:00 - H:M:S=0082:31:58 AVG= 0.83 DLT= 1.73
famous_w0oo_599_200_006747576 - PH 1 TS 0358021 A - 01/03/0638 01:00 - H:M:S=0082:36:55 AVG= 0.83 DLT= 1.75
famous_w0oo_599_200_006747576 - PH 1 TS 0358411 A - 16/03/0638 01:00 - H:M:S=0082:41:54 AVG= 0.83 DLT= 1.45
famous_w0oo_599_200_006747576 - PH 1 TS 0358801 A - 01/04/0638 01:00 - H:M:S=0082:46:50 AVG= 0.83 DLT= 1.72

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 31491
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 31713
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0359191 A - 16/04/0638 01:00 - H:M:S=0082:59:00 AVG= 0.83 DLT= 1.43
famous_w0oo_599_200_006747576 - PH 1 TS 0359581 A - 01/05/0638 01:00 - H:M:S=0083:03:58 AVG= 0.83 DLT= 1.42
famous_w0oo_599_200_006747576 - PH 1 TS 0359971 A - 16/05/0638 01:00 - H:M:S=0083:08:57 AVG= 0.83 DLT= 1.48
famous_w0oo_599_200_006747576 - PH 1 TS 0360361 A - 01/06/0638 01:00 - H:M:S=0083:13:55 AVG= 0.83 DLT= 1.60
famous_w0oo_599_200_006747576 - PH 1 TS 0360751 A - 16/06/0638 01:00 - H:M:S=0083:18:49 AVG= 0.83 DLT= 1.62
famous_w0oo_599_200_006747576 - PH 1 TS 0361141 A - 01/07/0638 01:00 - H:M:S=0083:23:52 AVG= 0.83 DLT= 1.68
famous_w0oo_599_200_006747576 - PH 1 TS 0361531 A - 16/07/0638 01:00 - H:M:S=0083:28:52 AVG= 0.83 DLT= 1.64
famous_w0oo_599_200_006747576 - PH 1 TS 0361921 A - 01/08/0638 01:00 - H:M:S=0083:33:49 AVG= 0.83 DLT= 1.70
famous_w0oo_599_200_006747576 - PH 1 TS 0362311 A - 16/08/0638 01:00 - H:M:S=0083:38:45 AVG= 0.83 DLT= 1.57

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 2362
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0362311 A - 16/08/0638 01:00 - H:M:S=0083:38:50 AVG= 0.83 DLT= 0.00

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 2417
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0362701 A - 01/09/0638 01:00 - H:M:S=0083:44:52 AVG= 0.83 DLT= 1.34

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 2825
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0363091 A - 16/09/0638 01:00 - H:M:S=0083:50:49 AVG= 0.83 DLT= 1.17
famous_w0oo_599_200_006747576 - PH 1 TS 0363481 A - 01/10/0638 01:00 - H:M:S=0083:55:51 AVG= 0.83 DLT= 1.89

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 3659
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0363871 A - 16/10/0638 01:00 - H:M:S=0084:02:15 AVG= 0.83 DLT= 1.63
CPDN Monitor - Quit request from BOINC...
Detaching shared memory...
Starting work on result famous_w0oo_599_200_006747576_2...
Starting model in /home/jsc/BOINC/projects/climateprediction.net...
Trying to kill old process # 24930
Trying to kill old process # 3659
Created shared memory region key = 129680 of size 743572 bytes (version 608)
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 4105
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0363871 A - 16/10/0638 01:00 - H:M:S=0084:02:15 AVG= 0.83 DLT= 0.00
famous_w0oo_599_200_006747576 - PH 1 TS 0364261 A - 01/11/0638 01:00 - H:M:S=0084:07:13 AVG= 0.83 DLT= 1.84
famous_w0oo_599_200_006747576 - PH 1 TS 0364651 A - 16/11/0638 01:00 - H:M:S=0084:12:12 AVG= 0.83 DLT= 1.46

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Program launched with process id # 4932
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Trickling yearly means for 638
Building file list for 638 in dataout -- 4 entries
Creating trickle file trickle_famous_w0oo_599_200_006747576_2_638.zip for upload...
famous_w0oo_599_200_006747576 - PH 1 TS 0365041 A - 01/12/0638 01:00 - H:M:S=0084:18:25 AVG= 0.83 DLT= 1.33

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 5300
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_w0oo_599_200_006747576 - PH 1 TS 0365431 A - 16/12/0638 01:00 - H:M:S=0084:24:03 AVG= 0.83 DLT= 1.38
famous_w0oo_599_200_006747576 - PH 1 TS 0365821 A - 01/01/0639 01:00 - H:M:S=0084:28:53 AVG= 0.83 DLT= 2.03

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 5996
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0365821 A - 01/01/0639 01:00 - H:M:S=0084:29:01 AVG= 0.83 DLT= 0.00
famous_w0oo_599_200_006747576 - PH 1 TS 0366211 A - 16/01/0639 01:00 - H:M:S=0084:33:54 AVG= 0.83 DLT= 1.67
famous_w0oo_599_200_006747576 - PH 1 TS 0366601 A - 01/02/0639 01:00 - H:M:S=0084:38:48 AVG= 0.83 DLT= 1.36

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 6972
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Program launched with process id # 7354
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 7719
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0366991 A - 16/02/0639 01:00 - H:M:S=0084:56:52 AVG= 0.83 DLT= 1.66
famous_w0oo_599_200_006747576 - PH 1 TS 0367381 A - 01/03/0639 01:00 - H:M:S=0085:01:50 AVG= 0.83 DLT= 1.28
famous_w0oo_599_200_006747576 - PH 1 TS 0367771 A - 16/03/0639 01:00 - H:M:S=0085:07:06 AVG= 0.83 DLT= 1.62
famous_w0oo_599_200_006747576 - PH 1 TS 0368161 A - 01/04/0639 01:00 - H:M:S=0085:11:56 AVG= 0.83 DLT= 1.18

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 9027
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 9395
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 9563
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0368551 A - 16/04/0639 01:00 - H:M:S=0085:27:03 AVG= 0.83 DLT= 1.73
famous_w0oo_599_200_006747576 - PH 1 TS 0368941 A - 01/05/0639 01:00 - H:M:S=0085:32:14 AVG= 0.83 DLT= 1.32
famous_w0oo_599_200_006747576 - PH 1 TS 0369331 A - 16/05/0639 01:00 - H:M:S=0085:37:22 AVG= 0.83 DLT= 2.15
famous_w0oo_599_200_006747576 - PH 1 TS 0369721 A - 01/06/0639 01:00 - H:M:S=0085:42:34 AVG= 0.83 DLT= 1.40
famous_w0oo_599_200_006747576 - PH 1 TS 0370111 A - 16/06/0639 01:00 - H:M:S=0085:47:43 AVG= 0.83 DLT= 1.48
famous_w0oo_599_200_006747576 - PH 1 TS 0370501 A - 01/07/0639 01:00 - H:M:S=0085:52:52 AVG= 0.83 DLT= 1.48

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 11952
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0370501 A - 01/07/0639 01:00 - H:M:S=0085:52:54 AVG= 0.83 DLT= 0.00
famous_w0oo_599_200_006747576 - PH 1 TS 0370891 A - 16/07/0639 01:00 - H:M:S=0085:58:05 AVG= 0.83 DLT= 2.12
famous_w0oo_599_200_006747576 - PH 1 TS 0371281 A - 01/08/0639 01:00 - H:M:S=0086:03:14 AVG= 0.83 DLT= 1.29
famous_w0oo_599_200_006747576 - PH 1 TS 0371671 A - 16/08/0639 01:00 - H:M:S=0086:08:19 AVG= 0.83 DLT= 1.45

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 13494
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0372061 A - 01/09/0639 01:00 - H:M:S=0086:18:24 AVG= 0.84 DLT= 1.31
famous_w0oo_599_200_006747576 - PH 1 TS 0372451 A - 16/09/0639 01:00 - H:M:S=0086:23:36 AVG= 0.84 DLT= 1.57
famous_w0oo_599_200_006747576 - PH 1 TS 0372841 A - 01/10/0639 01:00 - H:M:S=0086:28:48 AVG= 0.84 DLT= 1.81

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 14789
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0373231 A - 16/10/0639 01:00 - H:M:S=0086:34:26 AVG= 0.84 DLT= 1.96
famous_w0oo_599_200_006747576 - PH 1 TS 0373621 A - 01/11/0639 01:00 - H:M:S=0086:39:35 AVG= 0.84 DLT= 1.42
famous_w0oo_599_200_006747576 - PH 1 TS 0374011 A - 16/11/0639 01:00 - H:M:S=0086:44:47 AVG= 0.83 DLT= 1.34

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Trickling yearly means for 639
Building file list for 639 in dataout -- 4 entries
Creating trickle file trickle_famous_w0oo_599_200_006747576_2_639.zip for upload...
Processing decadal means for 630 through 639
Queuing intermediate upload for CPDN/BOINC: cpdnout4.zip
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 16519
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 16651
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_w0oo_599_200_006747576 - PH 1 TS 0374791 A - 16/12/0639 01:00 - H:M:S=0087:01:22 AVG= 0.84 DLT= 1.35
famous_w0oo_599_200_006747576 - PH 1 TS 0375181 A - 01/01/0640 01:00 - H:M:S=0087:06:34 AVG= 0.84 DLT= 1.63

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 17414
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0375571 A - 16/01/0640 01:00 - H:M:S=0087:12:06 AVG= 0.84 DLT= 1.52

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Program launched with process id # 18093
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
famous_w0oo_599_200_006747576 - PH 1 TS 0375571 A - 16/01/0640 01:00 - H:M:S=0087:12:08 AVG= 0.84 DLT= 0.00

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 18436
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0375961 A - 01/02/0640 01:00 - H:M:S=0087:21:35 AVG= 0.84 DLT= 1.32
famous_w0oo_599_200_006747576 - PH 1 TS 0376351 A - 16/02/0640 01:00 - H:M:S=0087:26:46 AVG= 0.84 DLT= 1.50

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 19444
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 19471
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0376741 A - 01/03/0640 01:00 - H:M:S=0087:34:27 AVG= 0.84 DLT= 1.43

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 19987
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0377131 A - 16/03/0640 01:00 - H:M:S=0087:41:51 AVG= 0.84 DLT= 1.17
famous_w0oo_599_200_006747576 - PH 1 TS 0377521 A - 01/04/0640 01:00 - H:M:S=0087:46:59 AVG= 0.84 DLT= 2.00
famous_w0oo_599_200_006747576 - PH 1 TS 0377911 A - 16/04/0640 01:00 - H:M:S=0087:52:14 AVG= 0.84 DLT= 1.44

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 21505
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0378301 A - 01/05/0640 01:00 - H:M:S=0088:00:10 AVG= 0.84 DLT= 1.63
famous_w0oo_599_200_006747576 - PH 1 TS 0378691 A - 16/05/0640 01:00 - H:M:S=0088:05:25 AVG= 0.84 DLT= 1.23

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 22454
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 22456
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 22689
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_w0oo_599_200_006747576 - PH 1 TS 0379081 A - 01/06/0640 01:00 - H:M:S=0088:15:08 AVG= 0.84 DLT= 1.47
famous_w0oo_599_200_006747576 - PH 1 TS 0379471 A - 16/06/0640 01:00 - H:M:S=0088:20:22 AVG= 0.84 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 23613
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 23916
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 24157
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 24189
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 5...
Starting model ID famous_w0oo_599_200_006747576   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 129680 /home/jsc/BOINC/slots/0
Program launched with process id # 24228
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
Model restart required...
Sorry, too many model crashes! :-(
Cleaning up from the run...
 (4103): called boinc_finish
Detaching shared memory...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
27 Nov 2010 07:18:03 922557 12174390 famous_w0oo_599_200_006747576_2 374,426 312,591 0.8349
26 Nov 2010 22:06:59 922557 12174390 famous_w0oo_599_200_006747576_2 365,066 303,505 0.8314
26 Nov 2010 17:34:19 922557 12174390 famous_w0oo_599_200_006747576_2 355,706 295,515 0.8308
26 Nov 2010 12:51:39 922557 12174390 famous_w0oo_599_200_006747576_2 346,346 286,975 0.8286
26 Nov 2010 08:43:15 922557 12174390 famous_w0oo_599_200_006747576_2 336,986 278,133 0.8254
26 Nov 2010 06:46:54 922557 12174390 famous_w0oo_599_200_006747576_2 327,626 269,193 0.8216
25 Nov 2010 21:09:11 922557 12174390 famous_w0oo_599_200_006747576_2 318,266 259,927 0.8167
25 Nov 2010 17:41:03 922557 12174390 famous_w0oo_599_200_006747576_2 308,906 251,988 0.8157
25 Nov 2010 14:59:14 922557 12174390 famous_w0oo_599_200_006747576_2 299,546 245,198 0.8186
25 Nov 2010 11:58:26 922557 12174390 famous_w0oo_599_200_006747576_2 290,186 237,545 0.8186
25 Nov 2010 09:14:33 922557 12174390 famous_w0oo_599_200_006747576_2 280,826 230,759 0.8217
25 Nov 2010 07:08:40 922557 12174390 famous_w0oo_599_200_006747576_2 271,466 223,724 0.8241
25 Nov 2010 07:08:39 922557 12174390 famous_w0oo_599_200_006747576_2 262,106 216,433 0.8257
24 Nov 2010 23:40:02 922557 12174390 famous_w0oo_599_200_006747576_2 252,746 208,889 0.8265
24 Nov 2010 19:09:07 922557 12174390 famous_w0oo_599_200_006747576_2 243,386 200,253 0.8228
24 Nov 2010 14:34:53 922557 12174390 famous_w0oo_599_200_006747576_2 234,026 191,402 0.8179
24 Nov 2010 09:54:56 922557 12174390 famous_w0oo_599_200_006747576_2 224,666 182,440 0.8120
24 Nov 2010 06:43:10 922557 12174390 famous_w0oo_599_200_006747576_2 215,306 173,136 0.8041
24 Nov 2010 06:43:10 922557 12174390 famous_w0oo_599_200_006747576_2 205,946 165,808 0.8051
23 Nov 2010 21:18:39 922557 12174390 famous_w0oo_599_200_006747576_2 196,586 157,421 0.8008


©2024 cpdn.org