climateprediction.net home page
Task 11728200

Task 11728200

Name famous_v640_1799_200_006693686_1
Workunit 6896939
Created 26 Aug 2010, 16:03:42 UTC
Sent 14 Dec 2010, 15:13:10 UTC
Report deadline 15 Mar 2011, 22:40:21 UTC
Received 17 Dec 2010, 15:07:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 922557
Run time
CPU time 14 hours 50 min 7 sec
Validate state Invalid
Credit 185.38
Device peak FLOPS 1.54 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-pc-linux-gnu
Stderr
<core_client_version>5.10.45</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
 AVG= 0.74 DLT= 1.45
famous_v640_1799_200_006693686 - PH 1 TS 0039391 A - 16/02/1804 01:00 - H:M:S=0008:08:26 AVG= 0.74 DLT= 1.68
famous_v640_1799_200_006693686 - PH 1 TS 0039781 A - 01/03/1804 01:00 - H:M:S=0008:12:58 AVG= 0.74 DLT= 1.78
famous_v640_1799_200_006693686 - PH 1 TS 0040171 A - 16/03/1804 01:00 - H:M:S=0008:17:32 AVG= 0.74 DLT= 1.41
famous_v640_1799_200_006693686 - PH 1 TS 0040561 A - 01/04/1804 01:00 - H:M:S=0008:22:08 AVG= 0.74 DLT= 1.65
famous_v640_1799_200_006693686 - PH 1 TS 0040951 A - 16/04/1804 01:00 - H:M:S=0008:26:41 AVG= 0.74 DLT= 1.55

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 21302
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0041341 A - 01/05/1804 01:00 - H:M:S=0008:34:22 AVG= 0.75 DLT= 1.46
famous_v640_1799_200_006693686 - PH 1 TS 0041731 A - 16/05/1804 01:00 - H:M:S=0008:38:51 AVG= 0.75 DLT= 1.42
famous_v640_1799_200_006693686 - PH 1 TS 0042121 A - 01/06/1804 01:00 - H:M:S=0008:43:20 AVG= 0.75 DLT= 1.57
famous_v640_1799_200_006693686 - PH 1 TS 0042511 A - 16/06/1804 01:00 - H:M:S=0008:47:52 AVG= 0.75 DLT= 1.74
famous_v640_1799_200_006693686 - PH 1 TS 0042901 A - 01/07/1804 01:00 - H:M:S=0008:52:25 AVG= 0.74 DLT= 1.25

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 22707
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0043291 A - 16/07/1804 01:00 - H:M:S=0008:58:37 AVG= 0.75 DLT= 1.42
famous_v640_1799_200_006693686 - PH 1 TS 0043681 A - 01/08/1804 01:00 - H:M:S=0009:03:09 AVG= 0.75 DLT= 1.25

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 23378
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 23711
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0044071 A - 16/08/1804 01:00 - H:M:S=0009:14:35 AVG= 0.76 DLT= 1.42
famous_v640_1799_200_006693686 - PH 1 TS 0044461 A - 01/09/1804 01:00 - H:M:S=0009:19:06 AVG= 0.75 DLT= 1.15
famous_v640_1799_200_006693686 - PH 1 TS 0044851 A - 16/09/1804 01:00 - H:M:S=0009:23:36 AVG= 0.75 DLT= 1.32

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 25198
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 25591
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
CPDN Monitor - Quit request from BOINC...
Detaching shared memory...
Starting work on result famous_v640_1799_200_006693686_1...
Starting model in /home/jsc/BOINC/projects/climateprediction.net...
Trying to kill old process # 12487
Trying to kill old process # 25591
Created shared memory region key = 132295 of size 743572 bytes (version 608)
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 25942
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 26321
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0045241 A - 01/10/1804 01:00 - H:M:S=0009:36:57 AVG= 0.77 DLT= 1.63
CPDN Monitor - Quit request from BOINC...
Detaching shared memory...
Starting work on result famous_v640_1799_200_006693686_1...
Starting model in /home/jsc/BOINC/projects/climateprediction.net...
Trying to kill old process # 25940
Trying to kill old process # 26321
Created shared memory region key = 132295 of size 743572 bytes (version 608)
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 27947
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0045241 A - 01/10/1804 01:00 - H:M:S=0009:36:57 AVG= 0.77 DLT= 0.00
famous_v640_1799_200_006693686 - PH 1 TS 0045631 A - 16/10/1804 01:00 - H:M:S=0009:41:29 AVG= 0.76 DLT= 1.17

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 28873
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0046021 A - 01/11/1804 01:00 - H:M:S=0009:49:12 AVG= 0.77 DLT= 1.66
famous_v640_1799_200_006693686 - PH 1 TS 0046411 A - 16/11/1804 01:00 - H:M:S=0009:53:39 AVG= 0.77 DLT= 1.26

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 30229
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0046411 A - 16/11/1804 01:00 - H:M:S=0009:53:51 AVG= 0.77 DLT= 0.00

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Trickling yearly means for 1804
Building file list for 1804 in dataout -- 4 entries
Creating trickle file trickle_famous_v640_1799_200_006693686_1_1804.zip for upload...
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 31274
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 31645
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
famous_v640_1799_200_006693686 - PH 1 TS 0047191 A - 16/12/1804 01:00 - H:M:S=0010:07:59 AVG= 0.77 DLT= 1.38
famous_v640_1799_200_006693686 - PH 1 TS 0047581 A - 01/01/1805 01:00 - H:M:S=0010:12:31 AVG= 0.77 DLT= 1.15
famous_v640_1799_200_006693686 - PH 1 TS 0047971 A - 16/01/1805 01:00 - H:M:S=0010:17:02 AVG= 0.77 DLT= 1.55

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 1158
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 1296
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0048361 A - 01/02/1805 01:00 - H:M:S=0010:22:53 AVG= 0.77 DLT= 1.33

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 2186
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0048751 A - 16/02/1805 01:00 - H:M:S=0010:28:25 AVG= 0.77 DLT= 1.54
famous_v640_1799_200_006693686 - PH 1 TS 0049141 A - 01/03/1805 01:00 - H:M:S=0010:33:01 AVG= 0.77 DLT= 1.37

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 3533
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0049141 A - 01/03/1805 01:00 - H:M:S=0010:33:09 AVG= 0.77 DLT= 0.00
famous_v640_1799_200_006693686 - PH 1 TS 0049531 A - 16/03/1805 01:00 - H:M:S=0010:37:42 AVG= 0.77 DLT= 1.32
famous_v640_1799_200_006693686 - PH 1 TS 0049921 A - 01/04/1805 01:00 - H:M:S=0010:42:15 AVG= 0.77 DLT= 1.57

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 4999
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 5342
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0050311 A - 16/04/1805 01:00 - H:M:S=0010:49:32 AVG= 0.77 DLT= 1.56
famous_v640_1799_200_006693686 - PH 1 TS 0050701 A - 01/05/1805 01:00 - H:M:S=0010:54:08 AVG= 0.77 DLT= 1.53
famous_v640_1799_200_006693686 - PH 1 TS 0051091 A - 16/05/1805 01:00 - H:M:S=0010:58:45 AVG= 0.77 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 7773
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 8025
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0051481 A - 01/06/1805 01:00 - H:M:S=0011:08:05 AVG= 0.78 DLT= 1.23
famous_v640_1799_200_006693686 - PH 1 TS 0051871 A - 16/06/1805 01:00 - H:M:S=0011:12:42 AVG= 0.78 DLT= 1.41
famous_v640_1799_200_006693686 - PH 1 TS 0052261 A - 01/07/1805 01:00 - H:M:S=0011:17:18 AVG= 0.78 DLT= 1.68

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 10225
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 10284
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0052651 A - 16/07/1805 01:00 - H:M:S=0011:24:27 AVG= 0.78 DLT= 1.49
famous_v640_1799_200_006693686 - PH 1 TS 0053041 A - 01/08/1805 01:00 - H:M:S=0011:29:02 AVG= 0.78 DLT= 1.42

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 11749
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0053431 A - 16/08/1805 01:00 - H:M:S=0011:34:32 AVG= 0.78 DLT= 1.10

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 12471
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0053821 A - 01/09/1805 01:00 - H:M:S=0011:39:58 AVG= 0.78 DLT= 1.30

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 13215
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0054211 A - 16/09/1805 01:00 - H:M:S=0011:45:46 AVG= 0.78 DLT= 1.65
famous_v640_1799_200_006693686 - PH 1 TS 0054601 A - 01/10/1805 01:00 - H:M:S=0011:50:26 AVG= 0.78 DLT= 1.37
famous_v640_1799_200_006693686 - PH 1 TS 0054991 A - 16/10/1805 01:00 - H:M:S=0011:55:03 AVG= 0.78 DLT= 1.45

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 15429
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0055381 A - 01/11/1805 01:00 - H:M:S=0012:01:54 AVG= 0.78 DLT= 1.60

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 16488
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0055771 A - 16/11/1805 01:00 - H:M:S=0012:08:05 AVG= 0.78 DLT= 1.45
Trickling yearly means for 1805
Building file list for 1805 in dataout -- 4 entries
Creating trickle file trickle_famous_v640_1799_200_006693686_1_1805.zip for upload...
famous_v640_1799_200_006693686 - PH 1 TS 0056161 A - 01/12/1805 01:00 - H:M:S=0012:12:37 AVG= 0.78 DLT= 1.70
famous_v640_1799_200_006693686 - PH 1 TS 0056551 A - 16/12/1805 01:00 - H:M:S=0012:17:07 AVG= 0.78 DLT= 1.23

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 19382
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 19947
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0056941 A - 01/01/1806 01:00 - H:M:S=0012:27:51 AVG= 0.79 DLT= 1.23
famous_v640_1799_200_006693686 - PH 1 TS 0057331 A - 16/01/1806 01:00 - H:M:S=0012:32:23 AVG= 0.79 DLT= 1.37

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 21372
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0057331 A - 16/01/1806 01:00 - H:M:S=0012:32:31 AVG= 0.79 DLT= 0.00

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 21840
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0057721 A - 01/02/1806 01:00 - H:M:S=0012:39:41 AVG= 0.79 DLT= 1.52
famous_v640_1799_200_006693686 - PH 1 TS 0058111 A - 16/02/1806 01:00 - H:M:S=0012:44:10 AVG= 0.79 DLT= 1.57

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 23988
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 24069
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 24460
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 24675
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0058501 A - 01/03/1806 01:00 - H:M:S=0012:56:43 AVG= 0.80 DLT= 1.45
famous_v640_1799_200_006693686 - PH 1 TS 0058891 A - 16/03/1806 01:00 - H:M:S=0013:01:13 AVG= 0.80 DLT= 1.18

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 26400
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0059281 A - 01/04/1806 01:00 - H:M:S=0013:07:07 AVG= 0.80 DLT= 1.48

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 27814
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 28043
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 28078
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 28786
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 5...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 29553
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0059671 A - 16/04/1806 01:00 - H:M:S=0013:25:49 AVG= 0.81 DLT= 1.51

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 31031
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 31617
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 31646
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0060061 A - 01/05/1806 01:00 - H:M:S=0013:37:06 AVG= 0.82 DLT= 1.70

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 32473
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 32694
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0060451 A - 16/05/1806 01:00 - H:M:S=0013:43:38 AVG= 0.82 DLT= 1.32
famous_v640_1799_200_006693686 - PH 1 TS 0060841 A - 01/06/1806 01:00 - H:M:S=0013:48:15 AVG= 0.82 DLT= 1.53
famous_v640_1799_200_006693686 - PH 1 TS 0061231 A - 16/06/1806 01:00 - H:M:S=0013:52:53 AVG= 0.82 DLT= 1.33

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 3205
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 3540
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 3731
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 3993
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
famous_v640_1799_200_006693686 - PH 1 TS 0061621 A - 01/07/1806 01:00 - H:M:S=0014:06:16 AVG= 0.82 DLT= 1.30
famous_v640_1799_200_006693686 - PH 1 TS 0062011 A - 16/07/1806 01:00 - H:M:S=0014:10:51 AVG= 0.82 DLT= 1.49

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 5481
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0062011 A - 16/07/1806 01:00 - H:M:S=0014:10:58 AVG= 0.82 DLT= 0.00
famous_v640_1799_200_006693686 - PH 1 TS 0062401 A - 01/08/1806 01:00 - H:M:S=0014:15:36 AVG= 0.82 DLT= 1.24
famous_v640_1799_200_006693686 - PH 1 TS 0062791 A - 16/08/1806 01:00 - H:M:S=0014:20:15 AVG= 0.82 DLT= 1.25

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Program launched with process id # 7291
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 7670
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.
famous_v640_1799_200_006693686 - PH 1 TS 0063181 A - 01/09/1806 01:00 - H:M:S=0014:28:49 AVG= 0.83 DLT= 1.22
famous_v640_1799_200_006693686 - PH 1 TS 0063571 A - 16/09/1806 01:00 - H:M:S=0014:33:29 AVG= 0.82 DLT= 1.35
famous_v640_1799_200_006693686 - PH 1 TS 0063961 A - 01/10/1806 01:00 - H:M:S=0014:38:09 AVG= 0.82 DLT= 1.41

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 1...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 10645
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 2...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 10880
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 3...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 10965
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 4...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 11261
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Model restart required...
Preparing for restart attempt # 5...
Starting model ID famous_v640_1799_200_006693686   Phase 1
Getting pthread attributes - retval=0
Setting pthread size (62914560 bytes) - retval=0
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu
Executing program /home/jsc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu 132295 /home/jsc/BOINC/slots/0
Program launched with process id # 11743
Climate model starting - use graphics to monitor progress.
Or visit the website to see the graphs for this run.

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
Model restart required...
Sorry, too many model crashes! :-(
Cleaning up from the run...
 (27945): called boinc_finish
Detaching shared memory...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
16 Dec 2010 09:40:06 922557 11728200 famous_v640_1799_200_006693686_1 56,186 43,957 0.7823
16 Dec 2010 09:40:06 922557 11728200 famous_v640_1799_200_006693686_1 46,826 35,901 0.7667
15 Dec 2010 20:24:50 922557 11728200 famous_v640_1799_200_006693686_1 37,466 27,821 0.7426
15 Dec 2010 17:12:23 922557 11728200 famous_v640_1799_200_006693686_1 28,106 21,067 0.7496
15 Dec 2010 13:04:39 922557 11728200 famous_v640_1799_200_006693686_1 18,746 13,998 0.7467
15 Dec 2010 10:26:43 922557 11728200 famous_v640_1799_200_006693686_1 9,386 7,011 0.7470


©2024 cpdn.org