Task 17815348

Name	hadam3pm2_k2ts_1959_10_009464891_1
Workunit	9547125
Created	22 Jan 2015, 17:02:20 UTC
Sent	22 Jan 2015, 22:28:27 UTC
Report deadline	5 Jan 2016, 3:48:27 UTC
Received	16 Feb 2015, 17:34:36 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1351624
Run time	3 days 1 hours 41 min 25 sec
CPU time	2 days 8 hours 14 min 23 sec
Validate state	Invalid
Credit	1,415.68
Device peak FLOPS	3.00 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0f39400] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8471d82] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8304353] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8073d03] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x808dc3c] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf0d19a83] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=887, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Stack trace (11 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0f72400] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x80b4e9b] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x80b1778] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81105a3] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x82c8fda] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x809011d] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf0d51a83] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (12 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0f84400] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x80f0b52] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8147bb6] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81c1a69] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8250b34] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8081970] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf0d63a83] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... SIGILL: illegal instruction Stack trace (12 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0f14400] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x80f0b12] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8147bb6] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81c1a69] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8250b34] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8081970] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf0cf3a83] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Stack trace (12 frames): /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0f06400] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8278f8f] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81488a8] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81c1a69] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8250b34] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8081970] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /var/lib/boinc-client/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf0ce5a83] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=26222, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 17:34:20 (26222): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Jan 2015 04:41:43	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	49,028	166,803	3.4022
29 Jan 2015 00:58:21	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	46,148	156,988	3.4018
28 Jan 2015 20:31:26	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	43,268	147,197	3.4020
28 Jan 2015 15:04:09	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	40,388	137,480	3.4040
28 Jan 2015 10:42:31	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	37,508	127,746	3.4058
24 Jan 2015 04:10:28	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	11,588	39,147	3.3782
24 Jan 2015 01:01:44	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	8,708	29,411	3.3775
23 Jan 2015 21:51:01	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	5,828	19,737	3.3866
23 Jan 2015 18:50:15	1351624	17815348	hadam3pm2_k2ts_1959_10_009464891_1	2,948	10,078	3.4186