Task 15561280

Name	hadcm3n_3b6u_1940_40_008265687_2
Workunit	8420811
Created	30 Jan 2013, 11:20:21 UTC
Sent	30 Jan 2013, 11:20:29 UTC
Report deadline	1 May 2013, 18:47:40 UTC
Received	5 Feb 2013, 10:52:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1170335
Run time	5 days 16 hours 30 min 3 sec
CPU time	5 days 12 hours 25 min 42 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:51:08 (32018): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:51:59 (11375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:54:21 (11449): No heartbeat from core client for 30 sec - exiting 10:54:25 (11449): No heartbeat from core client for 30 sec - exiting 10:54:26 (11449): No heartbeat from core client for 30 sec - exiting 10:54:27 (11449): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:56:05 (11517): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:56:07 (11517): No heartbeat from core client for 30 sec - exiting 10:56:08 (11517): No heartbeat from core client for 30 sec - exiting 10:58:05 (11567): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:06 (11618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:07 (11618): No heartbeat from core client for 30 sec - exiting 11:00:08 (11618): No heartbeat from core client for 30 sec - exiting 11:00:09 (11618): No heartbeat from core client for 30 sec - exiting 11:00:10 (11618): No heartbeat from core client for 30 sec - exiting 11:00:11 (11618): No heartbeat from core client for 30 sec - exiting 11:12:11 (11666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:17 (11666): No heartbeat from core client for 30 sec - exiting 11:19:07 (11713): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:13 (11713): No heartbeat from core client for 30 sec - exiting 11:19:14 (11713): No heartbeat from core client for 30 sec - exiting 11:19:15 (11713): No heartbeat from core client for 30 sec - exiting 11:19:16 (11713): No heartbeat from core client for 30 sec - exiting 11:24:41 (11763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:25:58 (11800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:00 (11800): No heartbeat from core client for 30 sec - exiting 11:26:01 (11800): No heartbeat from core client for 30 sec - exiting 11:27:12 (11836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:29:31 (11874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:30:52 (11910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:32:12 (11948): No heartbeat from core client for 30 sec - exiting 11:32:14 (11948): No heartbeat from core client for 30 sec - exiting 11:32:15 (11948): No heartbeat from core client for 30 sec - exiting 11:32:16 (11948): No heartbeat from core client for 30 sec - exiting 11:32:17 (11948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:28 (11978): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:30 (11978): No heartbeat from core client for 30 sec - exiting 11:34:31 (11978): No heartbeat from core client for 30 sec - exiting 11:40:52 (12020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77de400] [0xf77de430] /lib32/libc.so.6(gsignal+0x51)[0xf765f7d1] /lib32/libc.so.6(abort+0x182)[0xf7662c32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf764bb56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12063, iMonCtr=1 Model crash detected, will try to restart... 11:42:59 (12063): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:13 (12090): No heartbeat from core client for 30 sec - exiting 11:44:15 (12090): No heartbeat from core client for 30 sec - exiting 11:44:16 (12090): No heartbeat from core client for 30 sec - exiting 11:44:17 (12090): No heartbeat from core client for 30 sec - exiting 11:44:18 (12090): No heartbeat from core client for 30 sec - exiting 11:44:19 (12090): No heartbeat from core client for 30 sec - exiting 11:44:20 (12090): No heartbeat from core client for 30 sec - exiting 11:44:21 (12090): No heartbeat from core client for 30 sec - exiting 11:44:22 (12090): No heartbeat from core client for 30 sec - exiting 11:44:23 (12090): No heartbeat from core client for 30 sec - exiting 11:44:25 (12090): No heartbeat from core client for 30 sec - exiting 11:44:26 (12090): No heartbeat from core client for 30 sec - exiting 11:44:27 (12090): No heartbeat from core client for 30 sec - exiting 11:44:28 (12090): No heartbeat from core client for 30 sec - exiting 11:44:29 (12090): No heartbeat from core client for 30 sec - exiting 11:44:30 (12090): No heartbeat from core client for 30 sec - exiting 11:44:31 (12090): No heartbeat from core client for 30 sec - exiting 11:44:32 (12090): No heartbeat from core client for 30 sec - exiting 11:44:33 (12090): No heartbeat from core client for 30 sec - exiting 11:44:34 (12090): No heartbeat from core client for 30 sec - exiting 11:44:35 (12090): No heartbeat from core client for 30 sec - exiting 11:44:40 (12090): No heartbeat from core client for 30 sec - exiting 11:44:41 (12090): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77c6400] [0xf77c6430] /lib32/libc.so.6(gsignal+0x51)[0xf76477d1] /lib32/libc.so.6(abort+0x182)[0xf764ac32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7633b56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12122, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a3400] [0xf77a3430] /lib32/libc.so.6(gsignal+0x51)[0xf76247d1] /lib32/libc.so.6(abort+0x182)[0xf7627c32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7610b56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12122, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7761400] [0xf7761430] /lib32/libc.so.6(gsignal+0x51)[0xf75e27d1] /lib32/libc.so.6(abort+0x182)[0xf75e5c32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75ceb56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12122, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7734400] [0xf7734430] /lib32/libc.so.6(gsignal+0x51)[0xf75b57d1] /lib32/libc.so.6(abort+0x182)[0xf75b8c32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75a1b56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12122, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a3400] [0xf77a3430] /lib32/libc.so.6(gsignal+0x51)[0xf76247d1] /lib32/libc.so.6(abort+0x182)[0xf7627c32] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf7610b56] /home/antonio/BOINC/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12122, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Feb 2013 20:07:08	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	181,440	428,233	2.3602
04 Feb 2013 02:31:20	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	155,520	367,330	2.3619
03 Feb 2013 08:52:50	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	129,600	306,493	2.3649
02 Feb 2013 15:15:45	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	103,680	245,550	2.3683
01 Feb 2013 21:22:18	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	77,760	184,544	2.3733
01 Feb 2013 03:35:45	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	51,840	123,460	2.3816
31 Jan 2013 05:20:04	1170335	15561280	hadcm3n_3b6u_1940_40_008265687_2	25,920	61,547	2.3745