Task 15927103

Name	hadcm3n_4dtf_1980_40_008405711_0
Workunit	8556567
Created	20 Aug 2013, 3:24:59 UTC
Sent	20 Aug 2013, 3:31:43 UTC
Report deadline	19 Nov 2013, 10:58:54 UTC
Received	11 Oct 2013, 10:54:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1188779
Run time	14 days 4 hours 37 min 5 sec
CPU time	13 days 9 hours 47 min 18 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:06:59 (10992): No heartbeat from core client for 30 sec - exiting 09:07:00 (10992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:17:05 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:06 (4784): No heartbeat from core client for 30 sec - exiting 00:17:07 (4784): No heartbeat from core client for 30 sec - exiting 00:17:08 (4784): No heartbeat from core client for 30 sec - exiting 00:17:09 (4784): No heartbeat from core client for 30 sec - exiting 00:17:10 (4784): No heartbeat from core client for 30 sec - exiting 00:17:11 (4784): No heartbeat from core client for 30 sec - exiting 00:17:12 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=60064, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24200, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:25:55 (12944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:41 (2688): No heartbeat from core client for 30 sec - exiting 08:16:42 (2688): No heartbeat from core client for 30 sec - exiting 08:16:43 (2688): No heartbeat from core client for 30 sec - exiting 08:16:44 (2688): No heartbeat from core client for 30 sec - exiting 08:16:45 (2688): No heartbeat from core client for 30 sec - exiting 08:16:46 (2688): No heartbeat from core client for 30 sec - exiting 08:16:47 (2688): No heartbeat from core client for 30 sec - exiting 08:16:48 (2688): No heartbeat from core client for 30 sec - exiting 08:16:49 (2688): No heartbeat from core client for 30 sec - exiting 08:16:50 (2688): No heartbeat from core client for 30 sec - exiting 08:16:51 (2688): No heartbeat from core client for 30 sec - exiting 08:16:52 (2688): No heartbeat from core client for 30 sec - exiting 08:16:53 (2688): No heartbeat from core client for 30 sec - exiting 08:16:54 (2688): No heartbeat from core client for 30 sec - exiting 08:16:55 (2688): No heartbeat from core client for 30 sec - exiting 08:16:56 (2688): No heartbeat from core client for 30 sec - exiting 08:16:57 (2688): No heartbeat from core client for 30 sec - exiting 08:16:58 (2688): No heartbeat from core client for 30 sec - exiting 08:16:59 (2688): No heartbeat from core client for 30 sec - exiting 08:17:00 (2688): No heartbeat from core client for 30 sec - exiting 08:17:01 (2688): No heartbeat from core client for 30 sec - exiting 08:17:02 (2688): No heartbeat from core client for 30 sec - exiting 08:17:03 (2688): No heartbeat from core client for 30 sec - exiting 08:17:04 (2688): No heartbeat from core client for 30 sec - exiting 08:17:05 (2688): No heartbeat from core client for 30 sec - exiting 08:17:06 (2688): No heartbeat from core client for 30 sec - exiting 08:17:07 (2688): No heartbeat from core client for 30 sec - exiting 08:17:08 (2688): No heartbeat from core client for 30 sec - exiting 08:17:09 (2688): No heartbeat from core client for 30 sec - exiting 08:17:10 (2688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:58:12 (19656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:36:50 (5060): No heartbeat from core client for 30 sec - exiting 17:36:51 (5060): No heartbeat from core client for 30 sec - exiting 17:36:52 (5060): No heartbeat from core client for 30 sec - exiting 17:36:53 (5060): No heartbeat from core client for 30 sec - exiting 17:36:54 (5060): No heartbeat from core client for 30 sec - exiting 17:36:55 (5060): No heartbeat from core client for 30 sec - exiting 17:36:56 (5060): No heartbeat from core client for 30 sec - exiting 17:36:57 (5060): No heartbeat from core client for 30 sec - exiting 17:36:58 (5060): No heartbeat from core client for 30 sec - exiting 17:36:59 (5060): No heartbeat from core client for 30 sec - exiting 17:37:01 (5060): No heartbeat from core client for 30 sec - exiting 17:37:02 (5060): No heartbeat from core client for 30 sec - exiting 17:37:03 (5060): No heartbeat from core client for 30 sec - exiting 17:37:04 (5060): No heartbeat from core client for 30 sec - exiting 17:37:05 (5060): No heartbeat from core client for 30 sec - exiting 17:37:06 (5060): No heartbeat from core client for 30 sec - exiting 17:37:07 (5060): No heartbeat from core client for 30 sec - exiting 17:37:08 (5060): No heartbeat from core client for 30 sec - exiting 17:37:09 (5060): No heartbeat from core client for 30 sec - exiting 17:37:10 (5060): No heartbeat from core client for 30 sec - exiting 17:37:11 (5060): No heartbeat from core client for 30 sec - exiting 17:37:12 (5060): No heartbeat from core client for 30 sec - exiting 17:37:13 (5060): No heartbeat from core client for 30 sec - exiting 17:37:14 (5060): No heartbeat from core client for 30 sec - exiting 17:37:15 (5060): No heartbeat from core client for 30 sec - exiting 17:37:16 (5060): No heartbeat from core client for 30 sec - exiting 17:37:17 (5060): No heartbeat from core client for 30 sec - exiting 17:37:18 (5060): No heartbeat from core client for 30 sec - exiting 17:37:19 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:07:48 (13356): No heartbeat from core client for 30 sec - exiting 09:07:50 (13356): No heartbeat from core client for 30 sec - exiting 09:07:51 (13356): No heartbeat from core client for 30 sec - exiting 09:07:52 (13356): No heartbeat from core client for 30 sec - exiting 09:07:53 (13356): No heartbeat from core client for 30 sec - exiting 09:07:54 (13356): No heartbeat from core client for 30 sec - exiting 09:07:55 (13356): No heartbeat from core client for 30 sec - exiting 09:07:56 (13356): No heartbeat from core client for 30 sec - exiting 09:07:57 (13356): No heartbeat from core client for 30 sec - exiting 09:07:58 (13356): No heartbeat from core client for 30 sec - exiting 09:07:59 (13356): No heartbeat from core client for 30 sec - exiting 09:08:00 (13356): No heartbeat from core client for 30 sec - exiting 09:08:01 (13356): No heartbeat from core client for 30 sec - exiting 09:08:02 (13356): No heartbeat from core client for 30 sec - exiting 09:08:03 (13356): No heartbeat from core client for 30 sec - exiting 09:08:04 (13356): No heartbeat from core client for 30 sec - exiting 09:08:05 (13356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:45:46 (15876): No heartbeat from core client for 30 sec - exiting 17:45:48 (15876): No heartbeat from core client for 30 sec - exiting 17:45:49 (15876): No heartbeat from core client for 30 sec - exiting 17:45:50 (15876): No heartbeat from core client for 30 sec - exiting 17:45:51 (15876): No heartbeat from core client for 30 sec - exiting 17:45:52 (15876): No heartbeat from core client for 30 sec - exiting 17:45:53 (15876): No heartbeat from core client for 30 sec - exiting 17:45:54 (15876): No heartbeat from core client for 30 sec - exiting 17:45:55 (15876): No heartbeat from core client for 30 sec - exiting 17:45:56 (15876): No heartbeat from core client for 30 sec - exiting 17:45:57 (15876): No heartbeat from core client for 30 sec - exiting 17:45:58 (15876): No heartbeat from core client for 30 sec - exiting 17:45:59 (15876): No heartbeat from core client for 30 sec - exiting 17:46:00 (15876): No heartbeat from core client for 30 sec - exiting 17:46:01 (15876): No heartbeat from core client for 30 sec - exiting 17:46:02 (15876): No heartbeat from core client for 30 sec - exiting 17:46:03 (15876): No heartbeat from core client for 30 sec - exiting 17:46:04 (15876): No heartbeat from core client for 30 sec - exiting 17:46:05 (15876): No heartbeat from core client for 30 sec - exiting 17:46:06 (15876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Oct 2013 08:44:06	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	570,240	1,160,698	2.0355
29 Sep 2013 05:27:23	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	544,320	1,110,144	2.0395
23 Sep 2013 09:53:18	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	518,400	1,059,527	2.0438
20 Sep 2013 11:40:16	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	492,480	1,008,266	2.0473
09 Sep 2013 13:10:50	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	466,560	956,117	2.0493
08 Sep 2013 18:49:46	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	440,640	905,035	2.0539
07 Sep 2013 15:30:40	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	414,720	853,719	2.0585
06 Sep 2013 08:27:03	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	388,800	801,475	2.0614
05 Sep 2013 17:01:17	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	362,880	747,645	2.0603
04 Sep 2013 23:10:55	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	336,960	694,932	2.0624
04 Sep 2013 02:18:52	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	311,040	641,892	2.0637
03 Sep 2013 03:08:10	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	285,120	589,349	2.0670
02 Sep 2013 05:22:42	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	259,200	537,013	2.0718
01 Sep 2013 14:31:33	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	233,280	485,026	2.0792
31 Aug 2013 21:55:40	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	207,360	426,743	2.0580
31 Aug 2013 08:09:20	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	181,440	374,427	2.0636
30 Aug 2013 10:18:47	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	155,520	322,829	2.0758
29 Aug 2013 17:34:07	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	129,600	270,847	2.0899
29 Aug 2013 02:02:44	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	103,680	217,518	2.0980
28 Aug 2013 10:23:41	1188779	15927103	hadcm3n_4dtf_1980_40_008405711_0	77,760	164,365	2.1137