Task 17353110

Name	hadcm3n_x19t_1940_40_009148916_0
Workunit	9279252
Created	6 Nov 2014, 12:42:11 UTC
Sent	6 Nov 2014, 13:25:52 UTC
Report deadline	5 Feb 2015, 20:53:03 UTC
Received	26 Nov 2014, 10:38:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1338251
Run time	14 days 11 hours 24 min 44 sec
CPU time	9 days 5 hours 32 min 50 sec
Validate state	Invalid
Credit	10,264.32
Device peak FLOPS	3.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:49:52 (14356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:53 (14356): No heartbeat from core client for 30 sec - exiting 08:49:54 (14356): No heartbeat from core client for 30 sec - exiting 08:49:55 (14356): No heartbeat from core client for 30 sec - exiting 17:59:16 (33172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:45:10 (35284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:11 (35284): No heartbeat from core client for 30 sec - exiting 12:45:46 (36680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:17:17 (26260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:19 (26260): No heartbeat from core client for 30 sec - exiting 08:17:20 (26260): No heartbeat from core client for 30 sec - exiting 08:17:21 (26260): No heartbeat from core client for 30 sec - exiting 08:17:22 (26260): No heartbeat from core client for 30 sec - exiting 08:17:23 (26260): No heartbeat from core client for 30 sec - exiting 08:17:24 (26260): No heartbeat from core client for 30 sec - exiting 08:17:25 (26260): No heartbeat from core client for 30 sec - exiting 08:19:04 (39852): No heartbeat from core client for 30 sec - exiting 08:19:05 (39852): No heartbeat from core client for 30 sec - exiting 08:19:06 (39852): No heartbeat from core client for 30 sec - exiting 08:19:07 (39852): No heartbeat from core client for 30 sec - exiting 08:19:08 (39852): No heartbeat from core client for 30 sec - exiting 08:19:09 (39852): No heartbeat from core client for 30 sec - exiting 08:19:10 (39852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:45:01 (22268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:56 (47068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:57 (47068): No heartbeat from core client for 30 sec - exiting 01:07:58 (47068): No heartbeat from core client for 30 sec - exiting 01:07:59 (47068): No heartbeat from core client for 30 sec - exiting 01:08:00 (47068): No heartbeat from core client for 30 sec - exiting 01:08:01 (47068): No heartbeat from core client for 30 sec - exiting 02:00:05 (42756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:00:06 (42756): No heartbeat from core client for 30 sec - exiting 02:00:07 (42756): No heartbeat from core client for 30 sec - exiting 02:00:08 (42756): No heartbeat from core client for 30 sec - exiting 02:00:09 (42756): No heartbeat from core client for 30 sec - exiting 02:49:47 (45336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:48 (45336): No heartbeat from core client for 30 sec - exiting 02:49:49 (45336): No heartbeat from core client for 30 sec - exiting 02:49:50 (45336): No heartbeat from core client for 30 sec - exiting 03:13:56 (35636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:15:20 (41400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:18:26 (45860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:18:27 (45860): No heartbeat from core client for 30 sec - exiting 03:18:29 (45860): No heartbeat from core client for 30 sec - exiting 03:18:30 (45860): No heartbeat from core client for 30 sec - exiting 03:18:31 (45860): No heartbeat from core client for 30 sec - exiting 03:19:06 (47076): No heartbeat from core client for 30 sec - exiting 03:19:07 (47076): No heartbeat from core client for 30 sec - exiting 03:19:08 (47076): No heartbeat from core client for 30 sec - exiting 03:19:09 (47076): No heartbeat from core client for 30 sec - exiting 03:19:10 (47076): No heartbeat from core client for 30 sec - exiting 03:19:12 (47076): No heartbeat from core client for 30 sec - exiting 03:19:13 (47076): No heartbeat from core client for 30 sec - exiting 03:19:14 (47076): No heartbeat from core client for 30 sec - exiting 03:19:15 (47076): No heartbeat from core client for 30 sec - exiting 03:19:16 (47076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:13 (8944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:51:01 (12412): No heartbeat from core client for 30 sec - exiting 12:51:02 (12412): No heartbeat from core client for 30 sec - exiting 12:51:04 (12412): No heartbeat from core client for 30 sec - exiting 12:51:05 (12412): No heartbeat from core client for 30 sec - exiting 12:51:06 (12412): No heartbeat from core client for 30 sec - exiting 12:51:07 (12412): No heartbeat from core client for 30 sec - exiting 12:51:08 (12412): No heartbeat from core client for 30 sec - exiting 12:51:09 (12412): No heartbeat from core client for 30 sec - exiting 12:51:10 (12412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:19:48 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10504, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10996, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10996, iMonCtr=1 Model crash detected, will try to restart... 14:14:07 (10848): No heartbeat from core client for 30 sec - exiting 14:14:08 (10848): No heartbeat from core client for 30 sec - exiting 14:14:09 (10848): No heartbeat from core client for 30 sec - exiting 14:14:10 (10848): No heartbeat from core client for 30 sec - exiting 14:14:11 (10848): No heartbeat from core client for 30 sec - exiting 14:14:12 (10848): No heartbeat from core client for 30 sec - exiting 14:14:13 (10848): No heartbeat from core client for 30 sec - exiting 14:14:14 (10848): No heartbeat from core client for 30 sec - exiting 14:14:15 (10848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:14:16 (10848): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:28:47 (6472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:48 (6472): No heartbeat from core client for 30 sec - exiting 10:28:49 (6472): No heartbeat from core client for 30 sec - exiting 10:28:50 (6472): No heartbeat from core client for 30 sec - exiting 10:28:51 (6472): No heartbeat from core client for 30 sec - exiting 10:28:52 (6472): No heartbeat from core client for 30 sec - exiting 10:28:53 (6472): No heartbeat from core client for 30 sec - exiting 10:28:54 (6472): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:38:16 (11284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:49:01 (11032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Nov 2014 08:51:56	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	855,360	1,121,184	1.3108
25 Nov 2014 23:39:52	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	829,440	1,090,175	1.3144
25 Nov 2014 14:31:02	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	803,520	1,059,213	1.3182
25 Nov 2014 05:50:36	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	777,600	1,029,455	1.3239
24 Nov 2014 20:18:17	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	751,680	997,890	1.3275
24 Nov 2014 10:56:00	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	725,760	966,375	1.3315
24 Nov 2014 01:33:40	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	699,840	934,936	1.3359
23 Nov 2014 02:02:30	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	673,920	903,136	1.3401
22 Nov 2014 16:44:52	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	648,000	872,004	1.3457
22 Nov 2014 07:32:39	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	622,080	840,633	1.3513
21 Nov 2014 22:10:00	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	596,160	808,806	1.3567
21 Nov 2014 13:22:45	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	570,240	779,686	1.3673
21 Nov 2014 05:01:28	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	544,320	751,409	1.3805
20 Nov 2014 20:29:29	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	518,400	722,887	1.3945
20 Nov 2014 00:44:29	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	492,480	694,821	1.4109
19 Nov 2014 09:27:14	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	466,560	665,100	1.4255
19 Nov 2014 00:38:42	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	440,640	635,171	1.4415
18 Nov 2014 14:43:03	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	414,720	604,030	1.4565
18 Nov 2014 05:08:04	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	388,800	573,239	1.4744
17 Nov 2014 18:00:00	1338251	17353110	hadcm3n_x19t_1940_40_009148916_0	362,880	543,834	1.4987