Task 15524115

Name	hadcm3n_3e03_1940_40_008262099_1
Workunit	8417223
Created	6 Jan 2013, 5:54:59 UTC
Sent	6 Jan 2013, 5:55:34 UTC
Report deadline	7 Apr 2013, 13:22:45 UTC
Received	16 Jan 2013, 23:03:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1223462
Run time	4 days 13 hours 34 min 39 sec
CPU time	4 days 7 hours 18 min 32 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.84 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:45:53 (1961): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:28:23 (29700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d0400] /lib32/libm.so.6(sin+0x324)[0xf77496c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75916b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf773a400] /lib32/libm.so.6(sin+0x324)[0xf76b36c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74fb6b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... Model crashed: SETCONA:Error in locating levels for cloud lyrs tmp/pipe_dummy 2048 SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7780400] /lib32/libm.so.6(sin+0x324)[0xf76f96c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75416b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77d9400] /lib32/libm.so.6(sin+0x324)[0xf77526c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf759a6b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7776400] /lib32/libm.so.6(sin+0x324)[0xf76ef6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf75376b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... SIGSEGV: segmentation violation Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7734400] /lib32/libm.so.6(sin+0x324)[0xf76ad6c4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x822a4f8] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818a767] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x818d749] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8391957] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xf3)[0xf74f56b3] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1211, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jan 2013 19:17:44	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	259,200	346,279	1.3360
13 Jan 2013 10:14:38	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	233,280	311,589	1.3357
12 Jan 2013 15:50:45	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	207,360	276,820	1.3350
11 Jan 2013 10:24:34	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	181,440	241,408	1.3305
10 Jan 2013 17:11:03	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	155,520	206,922	1.3305
09 Jan 2013 14:05:22	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	129,600	172,794	1.3333
08 Jan 2013 13:35:21	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	103,680	138,554	1.3364
07 Jan 2013 20:22:49	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	77,760	103,979	1.3372
07 Jan 2013 03:19:50	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	51,840	69,295	1.3367
06 Jan 2013 15:42:40	1223462	15524115	hadcm3n_3e03_1940_40_008262099_1	25,920	34,648	1.3367