Task 15274626

Name	hadcm3n_zfyy_1880_40_008198719_0
Workunit	8353843
Created	12 Sep 2012, 20:17:18 UTC
Sent	15 Sep 2012, 6:31:33 UTC
Report deadline	15 Dec 2012, 13:58:44 UTC
Received	5 Oct 2012, 6:59:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1229231
Run time	18 days 20 hours 23 min 9 sec
CPU time	10 days 4 hours 28 min 39 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	0.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CreateFile error 32 when trying set file time Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3648, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... 12:43:54 (4236): No heartbeat from core client for 30 sec - exiting 12:43:55 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:00 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4736, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4736, iMonCtr=1 Model crash detected, will try to restart... C03:37:33 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:37:34 (3992): No heartbeat from core client for 30 sec - exiting 03:37:35 (3992): No heartbeat from core client for 30 sec - exiting 03:46:39 (908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3868, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2656, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6988, iMonCtr=1 Model crash detected, will try to restart... 23:14:17 (4372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:18 (4372): No heartbeat from core client for 30 sec - exiting 23:14:19 (4372): No heartbeat from core client for 30 sec - exiting 23:14:20 (4372): No heartbeat from core client for 30 sec - exiting 23:14:21 (4372): No heartbeat from core client for 30 sec - exiting 23:21:34 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:23:12 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:25 (4516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:32:14 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:32:15 (5712): No heartbeat from core client for 30 sec - exiting 23:40:39 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:40 (5656): No heartbeat from core client for 30 sec - exiting 23:58:32 (676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:11 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:46 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:11:47 (5712): No heartbeat from core client for 30 sec - exiting 00:11:48 (5712): No heartbeat from core client for 30 sec - exiting 00:15:33 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:34 (4232): No heartbeat from core client for 30 sec - exiting 00:29:45 (4164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:29:46 (4164): No heartbeat from core client for 30 sec - exiting 00:29:47 (4164): No heartbeat from core client for 30 sec - exiting 00:29:49 (4164): No heartbeat from core client for 30 sec - exiting 00:29:50 (4164): No heartbeat from core client for 30 sec - exiting 00:29:51 (4164): No heartbeat from core client for 30 sec - exiting 01:01:40 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:23 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:24 (5172): No heartbeat from core client for 30 sec - exiting 01:03:25 (5172): No heartbeat from core client for 30 sec - exiting 01:05:18 (6212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:05:19 (6212): No heartbeat from core client for 30 sec - exiting 01:09:51 (6880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:17:41 (6672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:37 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:38 (6632): No heartbeat from core client for 30 sec - exiting 01:22:39 (6632): No heartbeat from core client for 30 sec - exiting 01:26:12 (6840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:12 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:30 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:45:31 (3096): No heartbeat from core client for 30 sec - exiting 01:49:39 (3536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:40 (3536): No heartbeat from core client for 30 sec - exiting 01:49:41 (3536): No heartbeat from core client for 30 sec - exiting 01:53:05 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:06 (5656): No heartbeat from core client for 30 sec - exiting 01:55:16 (5536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:26 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:27 (5656): No heartbeat from core client for 30 sec - exiting 01:59:03 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:36 (5364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:13 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:47 (4292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:24 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:25 (5348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:44:51 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:48:43 (6708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:50:46 (4324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:50:47 (4324): No heartbeat from core client for 30 sec - exiting 03:57:54 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:21 (1944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:06 (2052): No heartbeat from core client for 30 sec - exiting 04:14:07 (2052): No heartbeat from core client for 30 sec - exiting 04:14:08 (2052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:18 (6524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:19 (6524): No heartbeat from core client for 30 sec - exiting 04:28:15 (1944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:16 (1944): No heartbeat from core client for 30 sec - exiting 04:28:17 (1944): No heartbeat from core client for 30 sec - exiting 04:28:18 (1944): No heartbeat from core client for 30 sec - exiting 04:28:19 (1944): No heartbeat from core client for 30 sec - exiting 04:28:20 (1944): No heartbeat from core client for 30 sec - exiting 04:32:46 (6960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:16 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:58:53 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:36 (4036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:50:05 (3280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 23:56:08 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 00:15:08 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:45 (760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:24:48 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:57 (6152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Oct 2012 11:58:50	1229231	15274626	hadcm3n_zfyy_1880_40_008198719_0	129,600	866,498	6.6859
30 Sep 2012 20:22:08	1229231	15274626	hadcm3n_zfyy_1880_40_008198719_0	103,680	693,831	6.6920
26 Sep 2012 22:31:27	1229231	15274626	hadcm3n_zfyy_1880_40_008198719_0	77,760	526,614	6.7723
23 Sep 2012 20:26:18	1229231	15274626	hadcm3n_zfyy_1880_40_008198719_0	51,840	352,084	6.7917
20 Sep 2012 02:30:13	1229231	15274626	hadcm3n_zfyy_1880_40_008198719_0	25,920	181,437	6.9999