Task 13774554

Name	hadcm3n_ykqj_1900_40_007514205_3
Workunit	7711680
Created	13 Dec 2011, 17:22:26 UTC
Sent	14 Dec 2011, 0:00:00 UTC
Report deadline	14 Mar 2012, 7:27:11 UTC
Received	28 Dec 2011, 14:14:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1033339
Run time	8 days 13 hours 15 min 18 sec
CPU time	6 days 8 hours 40 min 38 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.01 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13936, selfPID=13936, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/ykqjko.pja4c10 is not a valid UM file. Error converting file to netcdf: dataout/ykqjko.pja4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:25 (25864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:20:14 (16794): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:17 (16794): No heartbeat from core client for 30 sec - exiting 04:21:24 (29031): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:15 (29072): No heartbeat from core client for 30 sec - exiting 04:22:16 (29072): No heartbeat from core client for 30 sec - exiting 04:22:17 (29072): No heartbeat from core client for 30 sec - exiting 04:22:18 (29072): No heartbeat from core client for 30 sec - exiting 04:22:19 (29072): No heartbeat from core client for 30 sec - exiting 04:22:20 (29072): No heartbeat from core client for 30 sec - exiting 04:22:21 (29072): No heartbeat from core client for 30 sec - exiting 04:22:22 (29072): No heartbeat from core client for 30 sec - exiting 04:22:23 (29072): No heartbeat from core client for 30 sec - exiting 04:22:24 (29072): No heartbeat from core client for 30 sec - exiting 04:22:25 (29072): No heartbeat from core client for 30 sec - exiting 04:22:26 (29072): No heartbeat from core client for 30 sec - exiting 04:22:27 (29072): No heartbeat from core client for 30 sec - exiting 04:22:28 (29072): No heartbeat from core client for 30 sec - exiting 04:22:29 (29072): No heartbeat from core client for 30 sec - exiting 04:22:30 (29072): No heartbeat from core client for 30 sec - exiting 04:22:31 (29072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27315, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27315, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27315, iMonCtr=1 Model crash detected, will try to restart... 09:44:59 (27315): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:09 (27558): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:29 (27558): No heartbeat from core client for 30 sec - exiting 09:50:38 (27627): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:50:42 (27627): No heartbeat from core client for 30 sec - exiting 09:50:43 (27627):09:51:44 (27652): No heartbeat from core client for 30 sec - exiting 09:51:48 (27652): No heartbeat from core client for 30 sec - exiting 09:51:49 (27652): No heartbeat from core client for 30 sec - exiting 09:51:50 (27652): No heartbeat from core client for 30 sec - exiting 09:51:51 (27652): No heartbeat from core client for 30 sec - exiting 09:51:52 (27652): No heartbeat from core client for 30 sec - exiting 09:51:53 (27652): No heartbeat from core client for 30 sec - exiting 09:51:54 (27652): No heartbeat from core client for 30 sec - exiting 09:51:55 (27652): No heartbeat from core client for 30 sec - exiting 09:51:56 (27652): No heartbeat from core client for 30 sec - exiting 09:51:57 (27652): No heartbeat from core client for 30 sec - exiting 09:51:58 (27652): No heartbeat from core client for 30 sec - exiting 09:51:59 (27652): No heartbeat from core client for 30 sec - exiting 09:52:00 (27652): No heartbeat from core client for 30 sec - exiting 09:52:01 (27652): No heartbeat from core client for 30 sec - exiting 09:52:02 (27652): No heartbeat from core client for 30 sec - exiting 09:52:03 (27652): No heartbeat from core client for 30 sec - exiting 09:52:04 (27652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:05 (27652): No heartbeat from core client for 30 sec - exiting 09:52:10 (27652): No heartbeat from core client for 30 sec - exiting 09:52:11 (27652): No heartbeat from core client for 30 sec - exiting 09:53:52 (27677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:06 (27743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:29 (27768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:56:31 (27768): No heartbeat from core client for 30 sec - exiting 09:58:31 (27789): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:18 (27839): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:01:02 (27839): No heartbeat from core client for 30 sec - exiting 10:04:07 (27916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:05:32 (27983): No heartbeat from core client for 30 sec - exiting 10:05:35 (27983): No heartbeat from core client for 30 sec - exiting 10:05:36 (27983): No heartbeat from core client for 30 sec - exiting 10:05:37 (27983): No heartbeat from core client for 30 sec - exiting 10:05:38 (27983): No heartbeat from core client for 30 sec - exiting 10:05:39 (27983): No heartbeat from core client for 30 sec - exiting 10:05:40 (27983): No heartbeat from core client for 30 sec - exiting 10:05:41 (27983): No heartbeat from core client for 30 sec - exiting 10:05:42 (27983): No heartbeat from core client for 30 sec - exiting 10:05:43 (27983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:18 (28013): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:19 (28020): No heartbeat from core client for 30 sec - exiting 10:07:21 (28020): No heartbeat from core client for 30 sec - exiting 10:07:22 (28020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:10:25 (28062): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:15:25 (28098): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:53 (28193): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:55 (28193): No heartbeat from core client for 30 sec - exiting 10:22:48 (28237): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:56 (28260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:38 (28313): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:35:26 (28358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:53 (28419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:00 (28419): No heartbeat from core client for 30 sec - exiting 10:47:03 (28419): No heartbeat from core client for 30 sec - exiting 10:47:04 (28419): No heartbeat from core client for 30 sec - exiting 10:47:05 (28419): No heartbeat from core client for 30 sec - exiting 10:47:06 (28419): No heartbeat from core client for 30 sec - exiting 10:47:07 (28419): No heartbeat from core client for 30 sec - exiting 10:57:22 (28535): No heartbeat from core client for 30 sec - exiting 10:57:33 (28535): No heartbeat from core client for 30 sec - exiting 10:57:36 (28535): No heartbeat from core client for 30 sec - exiting 10:57:37 (28535): No heartbeat from core client for 30 sec - exiting 10:57:38 (28535): No heartbeat from core client for 30 sec - exiting 10:57:39 (28535): No heartbeat from core client for 30 sec - exiting 10:57:40 (28535): No heartbeat from core client for 30 sec - exiting 10:57:41 (28535): No heartbeat from core client for 30 sec - exiting 10:57:44 (28535): No heartbeat from core client for 30 sec - exiting 10:57:47 (28535): No heartbeat from core client for 30 sec - exiting 10:58:10 (28535): No heartbeat from core client for 30 sec - exiting 10:58:13 (28535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:58:14 (28535): No heartbeat from core client for 30 sec - exiting 10:58:15 (28535): No heartbeat from core client for 30 sec - exiting 10:58:16 (28535): No heartbeat from core client for 30 sec - exiting 11:00:13 (28641): No heartbeat from core client for 30 sec - exiting 11:01:08 (28641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:32 (28688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:34 (28688): No heartbeat from core client for 30 sec - exiting 11:05:18 (28743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28783, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28783, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28783, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 15:51:29 (28783): No heartbeat from core client for 30 sec - exiting Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 19:12:30	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	311,040	508,528	1.6349
25 Dec 2011 20:37:27	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	285,120	465,916	1.6341
24 Dec 2011 22:01:02	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	259,200	422,963	1.6318
23 Dec 2011 23:18:04	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	233,280	379,869	1.6284
23 Dec 2011 02:59:44	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	207,360	337,361	1.6269
22 Dec 2011 02:36:35	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	181,440	294,766	1.6246
20 Dec 2011 10:08:06	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	155,520	252,799	1.6255
19 Dec 2011 05:57:00	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	129,600	210,758	1.6262
18 Dec 2011 02:41:22	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	103,680	168,987	1.6299
16 Dec 2011 16:33:49	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	77,760	126,075	1.6213
15 Dec 2011 23:44:09	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	51,840	84,107	1.6224
15 Dec 2011 03:07:50	1033339	13774554	hadcm3n_ykqj_1900_40_007514205_3	25,920	42,021	1.6212