Task 18033897

Name	hadam3pm2_pe5b_1991_10_009528632_1
Workunit	9610375
Created	9 Mar 2015, 20:47:05 UTC
Sent	9 Mar 2015, 21:23:38 UTC
Report deadline	20 Feb 2016, 2:43:38 UTC
Received	30 Mar 2015, 19:19:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1337590
Run time	13 days 2 hours 51 min 58 sec
CPU time	7 days 21 hours 52 min 52 sec
Validate state	Invalid
Credit	8,068.48
Device peak FLOPS	1.59 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 01:12:06 (6421): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 00:46:46 (3191): called boinc_finish Signal 15 received, exiting... 00:46:46 (3191): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 23:39:38 (3029): called boinc_finish Signal 15 received, exiting... 23:39:38 (3029): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 00:44:56 (3195): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 01:40:37 (4631): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 01:11:45 (6526): called boinc_finish Signal 15 received, exiting... 01:11:45 (6526): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 21:51:38 (3058): called boinc_finish Signal 15 received, exiting... 21:51:38 (3058): called boinc_finish oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc oa.pc\|4dec.nc oa.pc\|5jan.nc oa.pc\|5feb.nc oa.pc\|5mar.nc oa.pc\|5apr.nc oa.pc\|5may.nc oa.pc\|5jun.nc oa.pc\|5jul.nc oa.pc\|5aug.nc oa.pc\|5sep.nc oa.pc\|5oct.nc oa.pc\|5nov.nc oa.pe\|4dec.nc oa.pe\|5jan.nc oa.pe\|5feb.nc oa.pe\|5mar.nc oa.pe\|5apr.nc oa.pe\|5may.nc oa.pe\|5jun.nc oa.pe\|5jul.nc oa.pe\|5aug.nc oa.pe\|5sep.nc oa.pe\|5oct.nc oa.pe\|5nov.nc Suspended CPDN Monitor - Suspend request from BOINC... SIGSEGV: segmentation violation Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2997, iMonCtr=1 Model crash detected, will try to restart... oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 23:12:39 (30269): called boinc_finish Signal 15 received, exiting... 23:12:40 (30269): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 17:55:05 (3341): called boinc_finish Signal 15 received, exiting... 17:55:05 (3341): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 23:48:14 (2892): called boinc_finish Signal 15 received, exiting... 23:48:14 (2892): called boinc_finish SIGSEGV: segmentation violation Stack trace (11 frames): /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x83bccdf] [0xf0fa6bc0] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8147b3f] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x81c1a69] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8250b34] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8081970] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x8094c0a] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x831e514] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x83205f9] /media/data/var/lib/boinc/projects/climateprediction.net/hadam3pm2_um_7.03_i686-pc-linux-gnu[0x834c82d] /usr/lib/libc.so.6(__libc_start_main+0xde)[0x42c57e7e] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2944, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 00:51:12 (2996): called boinc_finish Signal 15 received, exiting... 00:51:12 (2996): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 22:41:48 (3417): called boinc_finish Signal 15 received, exiting... 22:41:48 (3417): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 21:52:01 (3155): called boinc_finish Signal 15 received, exiting... 21:52:01 (3155): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 23:16:37 (3195): called boinc_finish Signal 15 received, exiting... 23:16:37 (3195): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 23:40:31 (4923): called boinc_finish Signal 15 received, exiting... 23:40:31 (4923): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 21:33:31 (3073): called boinc_finish Signal 15 received, exiting... 21:33:32 (3073): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 21:47:12 (3325): called boinc_finish Signal 15 received, exiting... 21:47:12 (3325): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... 22:45:31 (3569): called boinc_finish Signal 15 received, exiting... 22:45:31 (3569): called boinc_finish Signal 15 received, exiting... Signal 15 received, exiting... Model crashed: æM Model crashed: § Model crashed: § Model crashed: § Model crashed: § Model crashed: § Sorry, too many model crashes! :-( 21:18:33 (2623): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Mar 2015 09:33:22	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	279,428	588,724	2.1069
26 Mar 2015 09:27:26	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	256,388	504,045	1.9659
26 Mar 2015 09:26:37	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	253,508	493,239	1.9457
26 Mar 2015 09:25:22	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	250,628	482,223	1.9241
26 Mar 2015 09:23:16	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	247,748	471,341	1.9025
26 Mar 2015 09:23:16	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	244,868	460,441	1.8804
22 Mar 2015 21:54:48	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	213,188	370,969	1.7401
22 Mar 2015 19:14:11	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	210,308	361,010	1.7166
20 Mar 2015 19:37:11	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	175,748	236,333	1.3447
20 Mar 2015 13:48:39	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	169,988	215,058	1.2651
20 Mar 2015 10:47:15	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	167,108	204,378	1.2230
20 Mar 2015 08:46:41	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	164,228	193,822	1.1802
20 Mar 2015 05:40:05	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	161,348	183,201	1.1354
20 Mar 2015 02:42:02	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	158,468	172,583	1.0891
19 Mar 2015 23:19:34	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	155,588	162,360	1.0435
19 Mar 2015 20:18:49	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	152,708	552,586	3.6186
19 Mar 2015 17:17:43	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	149,828	541,987	3.6174
19 Mar 2015 14:51:57	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	146,948	531,435	3.6165
19 Mar 2015 11:46:07	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	144,068	520,918	3.6158
19 Mar 2015 08:44:52	1337590	18033897	hadam3pm2_pe5b_1991_10_009528632_1	141,188	510,439	3.6153