Task 13359854

Name	hadcm3n_o4b6_1940_40_007447732_1
Workunit	7645235
Created	9 Sep 2011, 18:54:45 UTC
Sent	9 Sep 2011, 19:18:42 UTC
Report deadline	10 Dec 2011, 2:45:53 UTC
Received	11 Oct 2011, 15:35:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1118718
Run time	30 days 23 hours 28 min 7 sec
CPU time	30 days 14 hours 45 min 9 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	0.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:37:25 (18271): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o4b6ko.pje8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pje8c10 Error: Input file: dataout/o4b6ko.pie8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pie8c10 Error: Input file: dataout/o4b6ko.pfe8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pfe8c10 Error: Input file: dataout/o4b6ka.phe8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.phe8c10 Error: Input file: dataout/o4b6ka.pge8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pge8c10 Error: Input file: dataout/o4b6ka.pee8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pee8c10 Error: Input file: dataout/o4b6ka.pde8c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pde8c10 06:25:59 (31704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:14:09 (31961): No heartbeat from core client for 30 sec - exiting 18:14:10 (31961): No heartbeat from core client for 30 sec - exiting 18:14:11 (31961): No heartbeat from core client for 30 sec - exiting 18:14:12 (31961): No heartbeat from core client for 30 sec - exiting 18:14:13 (31961): No heartbeat from core client for 30 sec - exiting 18:14:14 (31961): No heartbeat from core client for 30 sec - exiting 18:14:15 (31961): No heartbeat from core client for 30 sec - exiting 18:14:16 (31961): No heartbeat from core client for 30 sec - exiting 18:14:17 (31961): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o4b6ko.pjf4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pjf4c10 Error: Input file: dataout/o4b6ko.pif4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pif4c10 Error: Input file: dataout/o4b6ko.pff4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ko.pff4c10 Error: Input file: dataout/o4b6ka.phf4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.phf4c10 Error: Input file: dataout/o4b6ka.pgf4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pgf4c10 Error: Input file: dataout/o4b6ka.pef4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pef4c10 Error: Input file: dataout/o4b6ka.pdf4c10 is not a valid UM file. Error converting file to netcdf: dataout/o4b6ka.pdf4c10 Suspended CPDN Monitor - Suspend request from BOINC... 06:26:02 (9073): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:20:21 (14593): No heartbeat from core client for 30 sec - exiting 01:20:22 (14593): No heartbeat from core client for 30 sec - exiting 01:20:23 (14593): No heartbeat from core client for 30 sec - exiting 01:20:24 (14593): No heartbeat from core client for 30 sec - exiting 01:20:25 (14593): No heartbeat from core client for 30 sec - exiting 01:20:26 (14593): No heartbeat from core client for 30 sec - exiting 01:20:27 (14593): No heartbeat from core client for 30 sec - exiting 01:20:28 (14593): No heartbeat from core client for 30 sec - exiting 01:20:29 (14593): No heartbeat from core client for 30 sec - exiting 01:20:30 (14593): No heartbeat from core client for 30 sec - exiting 01:20:31 (14593): No heartbeat from core client for 30 sec - exiting 01:20:32 (14593): No heartbeat from core client for 30 sec - exiting 01:20:33 (14593): No heartbeat from core client for 30 sec - exiting 01:20:34 (14593): No heartbeat from core client for 30 sec - exiting 01:20:35 (14593): No heartbeat from core client for 30 sec - exiting 01:20:36 (14593): No heartbeat from core client for 30 sec - exiting 01:20:37 (14593): No heartbeat from core client for 30 sec - exiting 01:20:38 (14593): No heartbeat from core client for 30 sec - exiting 01:20:39 (14593): No heartbeat from core client for 30 sec - exiting 01:20:40 (14593): No heartbeat from core client for 30 sec - exiting 01:20:41 (14593): No heartbeat from core client for 30 sec - exiting 01:20:42 (14593): No heartbeat from core client for 30 sec - exiting 01:20:43 (14593): No heartbeat from core client for 30 sec - exiting 01:20:44 (14593): No heartbeat from core client for 30 sec - exiting 01:20:45 (14593): No heartbeat from core client for 30 sec - exiting 01:20:46 (14593): No heartbeat from core client for 30 sec - exiting 01:20:47 (14593): No heartbeat from core client for 30 sec - exiting 01:20:48 (14593): No heartbeat from core client for 30 sec - exiting 01:20:49 (14593): No heartbeat from core client for 30 sec - exiting 01:20:50 (14593): No heartbeat from core client for 30 sec - exiting 01:20:51 (14593): No heartbeat from core client for 30 sec - exiting 01:20:52 (14593): No heartbeat from core client for 30 sec - exiting 01:20:53 (14593): No heartbeat from core client for 30 sec - exiting 01:20:54 (14593): No heartbeat from core client for 30 sec - exiting 01:20:55 (14593): No heartbeat from core client for 30 sec - exiting 01:20:56 (14593): No heartbeat from core client for 30 sec - exiting 01:20:57 (14593): No heartbeat from core client for 30 sec - exiting 01:20:58 (14593): No heartbeat from core client for 30 sec - exiting 01:20:59 (14593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:21:00 (14593): No heartbeat from core client for 30 sec - exiting 01:54:58 (17401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:59 (17401): No heartbeat from core client for 30 sec - exiting 01:55:00 (17401): No heartbeat from core client for 30 sec - exiting 01:55:01 (17401): No heartbeat from core client for 30 sec - exiting 01:55:02 (17401): No heartbeat from core client for 30 sec - exiting 01:55:03 (17401): No heartbeat from core client for 30 sec - exiting 01:55:04 (17401): No heartbeat from core client for 30 sec - exiting 01:55:05 (17401): No heartbeat from core client for 30 sec - exiting 01:55:06 (17401): No heartbeat from core client for 30 sec - exiting 01:55:07 (17401): No heartbeat from core client for 30 sec - exiting 01:55:08 (17401): No heartbeat from core client for 30 sec - exiting 01:55:09 (17401): No heartbeat from core client for 30 sec - exiting 01:55:10 (17401): No heartbeat from core client for 30 sec - exiting 01:55:11 (17401): No heartbeat from core client for 30 sec - exiting 01:55:12 (17401): No heartbeat from core client for 30 sec - exiting 01:55:13 (17401): No heartbeat from core client for 30 sec - exiting 01:55:14 (17401): No heartbeat from core client for 30 sec - exiting 01:57:20 (17426): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:56 (17430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:00:32 (17434): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:07 (17441): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:42 (17445): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:18 (17449): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:06:53 (17456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:15 (17460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:11:14 (17467): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:13:14 (17471): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:14:25 (17475): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:00 (17482): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:35 (17489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:19:58 (17493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:35 (17500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:25:56 (17507): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:27:55 (17510): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:55 (17514): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:53 (17521): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:52 (17525): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:51 (17532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:37:51 (17536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:50 (17540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:41:49 (17547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:48 (17550): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:47 (17557): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:47:46 (17561): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:46 (17565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:45 (17572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:44 (17576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:08 (17583): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:09 (17583): No heartbeat from core client for 30 sec - exiting 03:10:10 (17583): No heartbeat from core client for 30 sec - exiting 03:10:11 (17583): No heartbeat from core client for 30 sec - exiting 03:10:12 (17583): No heartbeat from core client for 30 sec - exiting 03:10:13 (17583): No heartbeat from core client for 30 sec - exiting 03:10:14 (17583): No heartbeat from core client for 30 sec - exiting 03:10:15 (17583): No heartbeat from core client for 30 sec - exiting 03:10:16 (17583): No heartbeat from core client for 30 sec - exiting 03:10:17 (17583): No heartbeat from core client for 30 sec - exiting 03:10:18 (17583): No heartbeat from core client for 30 sec - exiting 03:10:19 (17583): No heartbeat from core client for 30 sec - exiting 03:10:20 (17583): No heartbeat from core client for 30 sec - exiting 03:10:21 (17583): No heartbeat from core client for 30 sec - exiting 03:10:22 (17583): No heartbeat from core client for 30 sec - exiting 03:10:23 (17583): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called 03:12:28 (17596): No heartbeat from core client for 30 sec - exiting Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb785b400] [0xb785b422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76e5651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76e8a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76d1bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:14:05 (17600): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77f0400] [0xb77f0422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb767a651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb767da82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7666bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:15:40 (17607): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb78c0400] [0xb78c0422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb774a651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb774da82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7736bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:17:15 (17611): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb783c400] [0xb783c422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76c6651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76c9a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76b2bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:18:51 (17618): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb778f400] [0xb778f422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7619651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb761ca82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7605bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:20:26 (17622): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7821400] [0xb7821422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76ab651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76aea82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7697bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... 03:22:01 (17629): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77c6400] [0xb77c6422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7650651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb7653a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb763cbd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb76ff400] [0xb76ff422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7589651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb758ca82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7575bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17633, iMonCtr=1 Model crash detected, will try to restart... 03:24:23 (17633): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7836400] [0xb7836422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76c0651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76c3a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76acbd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17641, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb78b5400] [0xb78b5422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb773f651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb7742a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb772bbd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17641, iMonCtr=1 Model crash detected, will try to restart... 03:26:00 (17641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb783d400] [0xb783d422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76c7651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76caa82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76b3bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17648, iMonCtr=1 Model crash detected, will try to restart... 03:27:35 (17648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77e6400] [0xb77e6422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7670651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb7673a82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb765cbd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17653, iMonCtr=1 Model crash detected, will try to restart... 03:29:10 (17653): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7872400] [0xb7872422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76fc651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76ffa82] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76e8bd6] /boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17661, iMonCtr=1 Model crash detected, will try to restart... 03:30:45 (17661): No heartbeat from core client for 30 sec - exiting Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2011 18:39:58	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	492,480	2,628,348	5.3370
09 Oct 2011 03:42:22	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	466,560	2,489,344	5.3355
07 Oct 2011 12:38:49	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	440,640	2,350,408	5.3341
05 Oct 2011 21:52:52	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	414,720	2,212,096	5.3340
04 Oct 2011 07:05:03	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	388,800	2,073,904	5.3341
02 Oct 2011 16:17:41	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	362,880	1,935,478	5.3337
01 Oct 2011 01:16:24	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	336,960	1,796,499	5.3315
29 Sep 2011 10:26:57	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	311,040	1,657,482	5.3288
27 Sep 2011 19:58:40	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	285,120	1,518,868	5.3271
26 Sep 2011 05:22:39	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	259,200	1,380,056	5.3243
24 Sep 2011 13:35:02	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	233,280	1,241,221	5.3207
22 Sep 2011 22:42:27	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	207,360	1,102,599	5.3173
21 Sep 2011 08:00:53	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	181,440	964,729	5.3171
19 Sep 2011 17:47:33	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	155,520	826,844	5.3166
18 Sep 2011 02:36:13	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	129,600	688,600	5.3133
16 Sep 2011 11:17:12	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	103,680	551,946	5.3236
14 Sep 2011 20:33:01	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	77,760	413,966	5.3236
13 Sep 2011 05:56:31	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	51,840	276,071	5.3254
11 Sep 2011 15:11:19	1118718	13359854	hadcm3n_o4b6_1940_40_007447732_1	25,920	138,005	5.3243