Task 15730629

Name	hadcm3n_zhqg_1920_40_008351655_0
Workunit	8502516
Created	17 Apr 2013, 23:23:45 UTC
Sent	17 Apr 2013, 23:26:42 UTC
Report deadline	18 Jul 2013, 6:53:53 UTC
Received	19 Jun 2013, 16:41:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1270788
Run time	40 days 8 hours 42 min 11 sec
CPU time	17 days 8 hours 1 min 29 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.49 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1582, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1582, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1582, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1582, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1582, iMonCtr=1 Model crash detected, will try to restart... 14:58:09 (1582): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:10 (9986): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:12:19 (10014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10176, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:58:21 (10176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:49:58 (12600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:00:52 (13132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:42 (13308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 00:36:06 (13368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:36:08 (13368): No heartbeat from core client for 30 sec - exiting 01:10:31 (15353): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /home/v3dmax/ÐÐ°Ð³ÑÑÐ·ÐºÐ¸/BOINC/projects/climateprediction.net/hadcm3n_zhqg_1920_40_008351655/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 May 2013 12:09:58	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	1,036,800	1,497,721	1.4446
27 May 2013 12:55:24	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	1,010,880	1,415,856	1.4006
26 May 2013 13:41:38	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	984,960	1,334,119	1.3545
25 May 2013 14:23:59	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	959,040	1,252,366	1.3059
24 May 2013 15:10:08	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	933,120	1,170,632	1.2545
23 May 2013 15:55:57	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	907,200	1,088,785	1.2002
22 May 2013 16:41:37	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	881,280	1,006,911	1.1426
21 May 2013 17:29:10	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	855,360	925,102	1.0815
20 May 2013 18:16:04	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	829,440	843,314	1.0167
19 May 2013 18:58:05	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	803,520	761,511	0.9477
18 May 2013 19:41:10	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	777,600	679,716	0.8741
17 May 2013 20:26:57	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	751,680	597,842	0.7953
16 May 2013 21:09:51	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	725,760	515,930	0.7109
15 May 2013 21:52:44	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	699,840	434,144	0.6203
14 May 2013 22:36:45	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	673,920	352,301	0.5228
13 May 2013 23:24:50	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	648,000	270,504	0.4174
13 May 2013 00:09:20	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	622,080	188,797	0.3035
12 May 2013 00:53:08	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	596,160	106,904	0.1793
11 May 2013 01:39:39	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	570,240	25,139	0.0441
09 May 2013 05:20:05	1270788	15730629	hadcm3n_zhqg_1920_40_008351655_0	544,320	1,523	0.0028