Task 13069647

Name	hadcm3n_t40o_1940_40_007312511_2
Workunit	7509941
Created	4 Jul 2011, 11:11:44 UTC
Sent	4 Jul 2011, 11:39:57 UTC
Report deadline	3 Oct 2011, 19:07:08 UTC
Received	9 Sep 2011, 19:25:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1149318
Run time	32 days 14 hours 49 min 29 sec
CPU time	31 days 2 hours 16 min 14 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 12:47:36 (27932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:48:26 (32611): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... 12:50:49 (3591): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:37 (8303): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:54:22 (15656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:23 (24103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:55:42 (27690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:57:24 (12373): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:01:36 (16022): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:18:46 (3765): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:21 (3791): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:30 (20081): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:51:57 (1023): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:10:52 (11223): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:16:22 (25052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:03 (3630): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:14:31 (15311): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:14:48 (22210): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:17:01 (15259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:17:30 (27824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:58 (31010): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t40oko.pjf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pjf6c10 Error: Input file: dataout/t40oko.pif6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pif6c10 Error: Input file: dataout/t40oko.pff6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pff6c10 Error: Input file: dataout/t40oka.phf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.phf6c10 Error: Input file: dataout/t40oka.pgf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.pgf6c10 Error: Input file: dataout/t40oka.pef6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.pef6c10 Error: Input file: dataout/t40oka.pdf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.pdf6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:22:55 (14392): No heartbeat from core client for 30 sec - exiting 17:23:01 (14392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:55:50 (28607): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:57:22 (31514): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:21:58 (13541): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:56 (13601): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:50:53 (4846): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:05:40 (9589): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9613, selfPID=9613, iMonCtr=1 17:58:24 (10706): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:54:38 (14057): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:05:46 (25631): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:30:34 (15468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:15:29 (18745): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:22 (23711): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:11 (28254): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:08 (29176): No heartbeat from core client for 30 sec - exiting 23:14:11 (29176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:24:07 (29853): No heartbeat from core client for 30 sec - exiting 23:26:46 (29853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t40oko.pjg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pjg7c10 Error: Input file: dataout/t40oko.pig7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pig7c10 Error: Input file: dataout/t40oko.pfg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oko.pfg7c10 Error: Input file: dataout/t40oka.phg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.phg7c10 Error: Input file: dataout/t40oka.pgg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.pgg7c10 Error: Input file: dataout/t40oka.peg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.peg7c10 Error: Input file: dataout/t40oka.pdg7c10 is not a valid UM file. Error converting file to netcdf: dataout/t40oka.pdg7c10 CPDN Monitor - Quit request from BOINC... 12:13:35 (5627): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:13:13 (9677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:13:28 (11433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:44 (19474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:16:49 (27691): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:15:53 (31589): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:17:11 (17478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:18:09 (5897): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:32:11 (19909): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:31:58 (27921): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:43 (28800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:52 (5775): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:34:08 (15994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:33:20 (18763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:35 (20552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:18 (23360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 15:36:28 (31795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:20:41 (22103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:43 (22103): No heartbeat from core client for 30 sec - exiting 20:20:44 (22103): No heartbeat from core client for 30 sec - exiting 20:20:45 (22103): No heartbeat from core client for 30 sec - exiting 20:20:46 (22103): No heartbeat from core client for 30 sec - exiting 20:20:47 (22103): No heartbeat from core client for 30 sec - exiting 20:20:48 (22103): No heartbeat from core client for 30 sec - exiting 20:20:49 (22103): No heartbeat from core client for 30 sec - exiting 20:20:50 (22103): No heartbeat from core client for 30 sec - exiting 20:20:51 (22103): No heartbeat from core client for 30 sec - exiting 20:20:52 (22103): No heartbeat from core client for 30 sec - exiting 20:20:53 (22103): No heartbeat from core client for 30 sec - exiting 20:20:54 (22103): No heartbeat from core client for 30 sec - exiting 20:20:55 (22103): No heartbeat from core client for 30 sec - exiting 20:20:56 (22103): No heartbeat from core client for 30 sec - exiting 20:20:57 (22103): No heartbeat from core client for 30 sec - exiting 20:20:58 (22103): No heartbeat from core client for 30 sec - exiting 20:20:59 (22103): No heartbeat from core client for 30 sec - exiting 20:21:00 (22103): No heartbeat from core client for 30 sec - exiting 20:21:01 (22103): No heartbeat from core client for 30 sec - exiting 20:21:02 (22103): No heartbeat from core client for 30 sec - exiting 20:21:03 (22103): No heartbeat from core client for 30 sec - exiting 20:21:04 (22103): No heartbeat from core client for 30 sec - exiting 20:21:05 (22103): No heartbeat from core client for 30 sec - exiting 20:21:06 (22103): No heartbeat from core client for 30 sec - exiting 20:21:07 (22103): No heartbeat from core client for 30 sec - exiting 20:21:08 (22103): No heartbeat from core client for 30 sec - exiting 20:21:09 (22103): No heartbeat from core client for 30 sec - exiting 20:21:10 (22103): No heartbeat from core client for 30 sec - exiting 20:21:11 (22103): No heartbeat from core client for 30 sec - exiting 20:21:12 (22103): No heartbeat from core client for 30 sec - exiting 20:21:13 (22103): No heartbeat from core client for 30 sec - exiting 20:21:14 (22103): No heartbeat from core client for 30 sec - exiting 20:21:15 (22103): No heartbeat from core client for 30 sec - exiting 20:21:16 (22103): No heartbeat from core client for 30 sec - exiting 20:21:17 (22103): No heartbeat from core client for 30 sec - exiting 20:21:18 (22103): No heartbeat from core client for 30 sec - exiting 20:21:19 (22103): No heartbeat from core client for 30 sec - exiting 20:21:20 (22103): No heartbeat from core client for 30 sec - exiting 20:21:21 (22103): No heartbeat from core client for 30 sec - exiting 20:21:22 (22103): No heartbeat from core client for 30 sec - exiting 20:21:23 (22103): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77b3400] [0xf77b3430] /lib32/libc.so.6(gsignal+0x51)[0xf761bea1] /lib32/libc.so.6(abort+0x17e)[0xf761f2ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf7607e37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77db400] [0xf77db430] /lib32/libc.so.6(gsignal+0x51)[0xf7643ea1] /lib32/libc.so.6(abort+0x17e)[0xf76472ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf762fe37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7717400] [0xf7717430] /lib32/libc.so.6(gsignal+0x51)[0xf757fea1] /lib32/libc.so.6(abort+0x17e)[0xf75832ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf756be37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76e9400] [0xf76e9430] /lib32/libc.so.6(gsignal+0x51)[0xf7551ea1] /lib32/libc.so.6(abort+0x17e)[0xf75552ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf753de37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf775d400] [0xf775d430] /lib32/libc.so.6(gsignal+0x51)[0xf75c5ea1] /lib32/libc.so.6(abort+0x17e)[0xf75c92ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75b1e37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf778e400] [0xf778e430] /lib32/libc.so.6(gsignal+0x51)[0xf75f6ea1] /lib32/libc.so.6(abort+0x17e)[0xf75fa2ce] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf75e2e37] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=967, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Sep 2011 01:05:20	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	933,120	2,623,960	2.8120
08 Sep 2011 03:21:27	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	907,200	2,549,592	2.8104
07 Sep 2011 05:17:45	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	881,280	2,475,245	2.8087
06 Sep 2011 07:39:21	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	855,360	2,400,894	2.8069
27 Aug 2011 08:17:36	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	829,440	2,327,348	2.8059
26 Aug 2011 12:57:27	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	803,520	2,259,586	2.8121
25 Aug 2011 15:13:51	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	777,600	2,184,593	2.8094
24 Aug 2011 17:57:44	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	751,680	2,111,879	2.8095
23 Aug 2011 20:14:03	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	725,760	2,037,544	2.8075
22 Aug 2011 22:48:52	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	699,840	1,963,329	2.8054
21 Aug 2011 23:37:11	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	673,920	1,889,155	2.8032
21 Aug 2011 01:39:35	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	648,000	1,814,616	2.8003
03 Aug 2011 17:58:34	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	622,080	1,741,556	2.7996
02 Aug 2011 16:23:28	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	596,160	1,666,929	2.7961
31 Jul 2011 21:44:22	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	570,240	1,595,130	2.7973
30 Jul 2011 08:03:08	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	544,320	1,518,905	2.7905
29 Jul 2011 06:08:53	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	518,400	1,443,874	2.7853
27 Jul 2011 18:10:47	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	492,480	1,370,049	2.7819
26 Jul 2011 04:24:11	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	466,560	1,299,760	2.7858
25 Jul 2011 22:47:30	1149318	13069647	hadcm3n_t40o_1940_40_007312511_2	440,640	1,228,674	2.7884