Task 15494815

Name	hadcm3n_3n4q_1940_40_008265613_0
Workunit	8420737
Created	21 Dec 2012, 13:15:46 UTC
Sent	22 Dec 2012, 19:11:44 UTC
Report deadline	24 Mar 2013, 2:38:55 UTC
Received	5 Jan 2013, 1:25:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1168797
Run time	4 days 17 hours 52 min 7 sec
CPU time	4 days 17 hours 52 min 7 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.95 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.4.5</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 SIGSEGV: segmentation violation Stack trace (13 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xffffe400] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8155092] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8159b01] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x81518bf] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x815a74a] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x8153291] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x807f0e3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x837e9f4] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839982e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f8b7] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/libc.so.6(__libc_start_main+0xfe)[0xf754ac0e] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jan 2013 19:22:45	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	311,040	399,725	1.2851
03 Jan 2013 21:27:06	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	285,120	353,705	1.2405
02 Jan 2013 23:43:56	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	259,200	310,354	1.1974
02 Jan 2013 03:54:05	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	233,280	260,883	1.1183
01 Jan 2013 01:19:46	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	207,360	216,890	1.0460
31 Dec 2012 05:42:51	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	181,440	167,792	0.9248
30 Dec 2012 08:04:16	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	155,520	125,896	0.8095
29 Dec 2012 16:12:25	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	129,600	76,691	0.5918
29 Dec 2012 02:18:44	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	103,680	118,657	1.1445
28 Dec 2012 12:30:12	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	77,760	69,537	0.8943
27 Dec 2012 20:50:45	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	51,840	20,443	0.3943
27 Dec 2012 03:23:17	1168797	15494815	hadcm3n_3n4q_1940_40_008265613_0	25,920	18,335	0.7074