Task 15674776

Name	hadcm3n_4bzz_1940_40_008307568_1
Workunit	8458703
Created	21 Mar 2013, 15:08:52 UTC
Sent	21 Mar 2013, 15:08:57 UTC
Report deadline	20 Jun 2013, 22:36:08 UTC
Received	3 Apr 2013, 20:41:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1072904
Run time	8 days 11 hours 3 min 19 sec
CPU time	6 days 6 hours 5 min 57 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	3.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 17:19:51 (12959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:03 (24601): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:04 (24601): No heartbeat from core client for 30 sec - exiting 17:31:05 (24601): No heartbeat from core client for 30 sec - exiting 17:31:06 (24601): No heartbeat from core client for 30 sec - exiting 17:31:07 (24601): No heartbeat from core client for 30 sec - exiting 17:31:08 (24601): No heartbeat from core client for 30 sec - exiting 17:31:09 (24601): No heartbeat from core client for 30 sec - exiting 17:31:10 (24601): No heartbeat from core client for 30 sec - exiting 17:31:11 (24601): No heartbeat from core client for 30 sec - exiting 17:31:12 (24601): No heartbeat from core client for 30 sec - exiting 17:31:13 (24601): No heartbeat from core client for 30 sec - exiting 17:31:14 (24601): No heartbeat from core client for 30 sec - exiting 17:31:15 (24601): No heartbeat from core client for 30 sec - exiting 17:31:16 (24601): No heartbeat from core client for 30 sec - exiting 17:31:17 (24601): No heartbeat from core client for 30 sec - exiting 17:31:18 (24601): No heartbeat from core client for 30 sec - exiting 17:31:19 (24601): No heartbeat from core client for 30 sec - exiting 17:31:20 (24601): No heartbeat from core client for 30 sec - exiting 17:31:21 (24601): No heartbeat from core client for 30 sec - exiting 17:31:22 (24601): No heartbeat from core client for 30 sec - exiting 17:31:23 (24601): No heartbeat from core client for 30 sec - exiting 17:31:24 (24601): No heartbeat from core client for 30 sec - exiting 17:31:25 (24601): No heartbeat from core client for 30 sec - exiting 17:31:26 (24601): No heartbeat from core client for 30 sec - exiting 17:31:27 (24601): No heartbeat from core client for 30 sec - exiting 17:31:28 (24601): No heartbeat from core client for 30 sec - exiting 17:31:29 (24601): No heartbeat from core client for 30 sec - exiting 17:31:30 (24601): No heartbeat from core client for 30 sec - exiting 17:31:31 (24601): No heartbeat from core client for 30 sec - exiting 17:31:32 (24601): No heartbeat from core client for 30 sec - exiting 17:31:33 (24601): No heartbeat from core client for 30 sec - exiting 17:31:34 (24601): No heartbeat from core client for 30 sec - exiting 17:31:35 (24601): No heartbeat from core client for 30 sec - exiting 17:31:36 (24601): No heartbeat from core client for 30 sec - exiting 17:31:37 (24601): No heartbeat from core client for 30 sec - exiting 17:31:38 (24601): No heartbeat from core client for 30 sec - exiting 17:31:39 (24601): No heartbeat from core client for 30 sec - exiting 17:31:40 (24601): No heartbeat from core client for 30 sec - exiting 17:31:41 (24601): No heartbeat from core client for 30 sec - exiting 17:31:42 (24601): No heartbeat from core client for 30 sec - exiting 17:31:43 (24601): No heartbeat from core client for 30 sec - exiting 17:31:44 (24601): No heartbeat from core client for 30 sec - exiting 17:31:45 (24601): No heartbeat from core client for 30 sec - exiting 17:31:46 (24601): No heartbeat from core client for 30 sec - exiting 17:31:47 (24601): No heartbeat from core client for 30 sec - exiting 17:31:48 (24601): No heartbeat from core client for 30 sec - exiting 17:31:49 (24601): No heartbeat from core client for 30 sec - exiting 17:31:50 (24601): No heartbeat from core client for 30 sec - exiting 17:31:51 (24601): No heartbeat from core client for 30 sec - exiting 17:31:52 (24601): No heartbeat from core client for 30 sec - exiting 17:31:53 (24601): No heartbeat from core client for 30 sec - exiting 17:31:54 (24601): No heartbeat from core client for 30 sec - exiting 17:31:55 (24601): No heartbeat from core client for 30 sec - exiting 17:31:56 (24601): No heartbeat from core client for 30 sec - exiting 17:31:57 (24601): No heartbeat from core client for 30 sec - exiting 17:31:58 (24601): No heartbeat from core client for 30 sec - exiting 17:31:59 (24601): No heartbeat from core client for 30 sec - exiting 17:32:00 (24601): No heartbeat from core client for 30 sec - exiting 02:00:48 (24686): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:04 (40171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:05 (40171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:43:25 (43230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:26 (43230): No heartbeat from core client for 30 sec - exiting 08:43:27 (43230): No heartbeat from core client for 30 sec - exiting 08:43:28 (43230): No heartbeat from core client for 30 sec - exiting 08:43:29 (43230): No heartbeat from core client for 30 sec - exiting 08:43:30 (43230): No heartbeat from core client for 30 sec - exiting 08:43:31 (43230): No heartbeat from core client for 30 sec - exiting 08:43:32 (43230): No heartbeat from core client for 30 sec - exiting 08:43:33 (43230): No heartbeat from core client for 30 sec - exiting 08:43:34 (43230): No heartbeat from core client for 30 sec - exiting 08:43:35 (43230): No heartbeat from core client for 30 sec - exiting 08:43:36 (43230): No heartbeat from core client for 30 sec - exiting 08:43:37 (43230): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x1033e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x844e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x860400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x1033e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x844e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x1001404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x83e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x83e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x83e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x1002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x1002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(192,0xa0c46540) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/4bzzko.pjf2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzko.pjf2c10 Error: Input file: dataout/4bzzko.pif2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzko.pif2c10 Error: Input file: dataout/4bzzko.pff2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzko.pff2c10 Error: Input file: dataout/4bzzka.phf2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzka.phf2c10 Error: Input file: dataout/4bzzka.pgf2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzka.pgf2c10 Error: Input file: dataout/4bzzka.pef2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzka.pef2c10 Error: Input file: dataout/4bzzka.pdf2c10 is not a valid UM file. Error converting file to netcdf: dataout/4bzzka.pdf2c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:52:25 (30325): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:26 (30325): No heartbeat from core client for 30 sec - exiting 23:52:27 (30325): No heartbeat from core client for 30 sec - exiting 07:46:18 (168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:19 (168): No heartbeat from core client for 30 sec - exiting 07:46:20 (168): No heartbeat from core client for 30 sec - exiting 07:46:21 (168): No heartbeat from core client for 30 sec - exiting 07:46:22 (168): No heartbeat from core client for 30 sec - exiting 07:46:23 (168): No heartbeat from core client for 30 sec - exiting 07:46:24 (168): No heartbeat from core client for 30 sec - exiting 07:46:25 (168): No heartbeat from core client for 30 sec - exiting 07:46:26 (168): No heartbeat from core client for 30 sec - exiting 07:46:27 (168): No heartbeat from core client for 30 sec - exiting 07:46:28 (168): No heartbeat from core client for 30 sec - exiting 07:46:29 (168): No heartbeat from core client for 30 sec - exiting 07:46:30 (168): No heartbeat from core client for 30 sec - exiting 07:46:31 (168): No heartbeat from core client for 30 sec - exiting 07:46:32 (168): No heartbeat from core client for 30 sec - exiting 07:46:33 (168): No heartbeat from core client for 30 sec - exiting 07:46:34 (168): No heartbeat from core client for 30 sec - exiting 07:46:35 (168): No heartbeat from core client for 30 sec - exiting 07:46:36 (168): No heartbeat from core client for 30 sec - exiting 07:46:37 (168): No heartbeat from core client for 30 sec - exiting 07:46:38 (168): No heartbeat from core client for 30 sec - exiting 07:46:39 (168): No heartbeat from core client for 30 sec - exiting 07:46:40 (168): No heartbeat from core client for 30 sec - exiting 07:46:41 (168): No heartbeat from core client for 30 sec - exiting 07:46:42 (168): No heartbeat from core client for 30 sec - exiting 07:46:43 (168): No heartbeat from core client for 30 sec - exiting 07:46:44 (168): No heartbeat from core client for 30 sec - exiting 07:46:45 (168): No heartbeat from core client for 30 sec - exiting 07:46:46 (168): No heartbeat from core client for 30 sec - exiting 07:46:47 (168): No heartbeat from core client for 30 sec - exiting 07:46:48 (168): No heartbeat from core client for 30 sec - exiting 07:46:49 (168): No heartbeat from core client for 30 sec - exiting 07:46:50 (168): No heartbeat from core client for 30 sec - exiting 07:46:51 (168): No heartbeat from core client for 30 sec - exiting 07:46:52 (168): No heartbeat from core client for 30 sec - exiting 07:46:53 (168): No heartbeat from core client for 30 sec - exiting 07:46:54 (168): No heartbeat from core client for 30 sec - exiting 07:46:55 (168): No heartbeat from core client for 30 sec - exiting 07:46:56 (168): No heartbeat from core client for 30 sec - exiting 07:46:57 (168): No heartbeat from core client for 30 sec - exiting 07:46:58 (168): No heartbeat from core client for 30 sec - exiting 07:46:59 (168): No heartbeat from core client for 30 sec - exiting 07:47:00 (168): No heartbeat from core client for 30 sec - exiting 07:47:01 (168): No heartbeat from core client for 30 sec - exiting 07:47:02 (168): No heartbeat from core client for 30 sec - exiting 07:47:03 (168): No heartbeat from core client for 30 sec - exiting 08:23:39 (2677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:42 (2677): No heartbeat from core client for 30 sec - exiting 08:23:43 (2677): No heartbeat from core client for 30 sec - exiting 08:29:45 (3089): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:29:46 (3089): No heartbeat from core client for 30 sec - exiting 08:29:47 (3089): No heartbeat from core client for 30 sec - exiting 08:30:04 (3089): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 3782656 Failed with Error Code -1 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Apr 2013 08:50:35	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	440,640	534,471	1.2129
02 Apr 2013 18:41:47	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	414,720	502,295	1.2112
01 Apr 2013 22:49:48	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	388,800	470,918	1.2112
01 Apr 2013 00:21:20	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	362,880	439,330	1.2107
31 Mar 2013 12:05:11	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	336,960	407,254	1.2086
30 Mar 2013 14:45:28	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	311,040	375,705	1.2079
29 Mar 2013 20:38:36	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	285,120	344,914	1.2097
28 Mar 2013 23:31:00	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	259,200	314,241	1.2123
28 Mar 2013 03:16:51	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	233,280	283,475	1.2152
26 Mar 2013 15:45:30	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	207,360	251,865	1.2146
25 Mar 2013 04:16:07	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	181,440	220,085	1.2130
24 Mar 2013 12:33:14	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	155,520	189,373	1.2177
23 Mar 2013 20:36:41	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	129,600	157,579	1.2159
23 Mar 2013 09:08:23	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	103,680	126,268	1.2179
22 Mar 2013 21:47:33	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	77,760	94,368	1.2136
22 Mar 2013 11:34:52	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	51,840	62,668	1.2089
22 Mar 2013 00:56:34	1072904	15674776	hadcm3n_4bzz_1940_40_008307568_1	25,920	31,202	1.2038