Task 13147969

Name	hadcm3n_y9p7_1900_40_007345429_2
Workunit	7542859
Created	20 Jul 2011, 11:21:34 UTC
Sent	20 Jul 2011, 11:41:46 UTC
Report deadline	19 Oct 2011, 19:08:57 UTC
Received	10 Aug 2011, 8:18:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1138119
Run time	13 days 6 hours 6 min 51 sec
CPU time	11 days 0 hours 29 min 4 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	0.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 20:42:48 (3294): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x1ba400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fd938] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f0c06] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8142ae4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f4942] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807e843] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (15 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x492400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x831d26c] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (17 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x385400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8066f7e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81ca51e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8309a29] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (20 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xd8f400] /lib/libm.so.6[0x84c559] /lib/libm.so.6(expf+0x7e)[0x858b7e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8339966] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82f3099] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x833e3fb] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8336d09] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x820b23a] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82aa92e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8140cfb] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81dadc8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8267a6b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807b49e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (15 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xe1d400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x831412b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x828d3b0] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8090808] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x808a417] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8299901] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x821c545] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8219da1] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838720b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83995c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xd25400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8302b30] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8071efc] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837738c] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (16 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x37c400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82c55da] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83374ba] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x820b23a] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82aa92e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8140cfb] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82f1357] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x82646ad] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807b49e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x6aee36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (14 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0x6a5400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fe8a9] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80fd19b] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f0c06] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8142ae4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x80f4942] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807e843] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xe6)[0x126e36] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2060, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Aug 2011 19:52:02	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	155,520	927,136	5.9615
05 Aug 2011 20:47:25	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	129,600	774,675	5.9774
30 Jul 2011 10:54:15	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	103,680	735,278	7.0918
27 Jul 2011 14:58:34	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	77,760	553,349	7.1161
25 Jul 2011 22:51:56	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	51,840	373,363	7.2022
25 Jul 2011 20:18:43	1138119	13147969	hadcm3n_y9p7_1900_40_007345429_2	25,920	188,466	7.2711