Task 19133211

Name	hadcm3n_sa1x_198012_480_010221804_0
Workunit	10221804
Created	4 Dec 2015, 19:22:26 UTC
Sent	5 Dec 2015, 3:43:47 UTC
Report deadline	16 Nov 2016, 9:03:47 UTC
Received	19 Mar 2016, 17:51:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1359488
Run time	2 days 7 hours 46 min
CPU time	1 days 20 hours 13 min 53 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	4.56 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 01:29:37 (276600): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:20:17 (89608): start_timer_thread(): CreateThread() failed, errno 0 12:22:34 (90000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:23:10 (95104): start_timer_thread(): CreateThread() failed, errno 0 12:24:32 (99928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:13 (100092): start_timer_thread(): CreateThread() failed, errno 0 12:27:08 (100896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:50 (98908): start_timer_thread(): CreateThread() failed, errno 0 15:05:45 (96896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:06:21 (104664): start_timer_thread(): CreateThread() failed, errno 0 15:10:15 (105244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:50 (106612): start_timer_thread(): CreateThread() failed, errno 0 15:11:38 (104616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:11 (106688): start_timer_thread(): CreateThread() failed, errno 0 15:12:48 (100908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:23 (93272): start_timer_thread(): CreateThread() failed, errno 0 15:14:00 (106520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:14:23 (100280): start_timer_thread(): CreateThread() failed, errno 0 15:14:47 (98920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:11 (99940): start_timer_thread(): CreateThread() failed, errno 0 15:16:48 (101224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:17:23 (100760): start_timer_thread(): CreateThread() failed, errno 0 15:17:54 (93424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:18:29 (104044): start_timer_thread(): CreateThread() failed, errno 0 15:18:59 (3192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:19:36 (95080): start_timer_thread(): CreateThread() failed, errno 0 15:20:44 (105868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:57 (85220): start_timer_thread(): CreateThread() failed, errno 0 15:21:26 (107076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:21:59 (95912): start_timer_thread(): CreateThread() failed, errno 0 15:22:24 (96976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:26 (99676): start_timer_thread(): CreateThread() failed, errno 0 15:23:56 (105964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:24:56 (90392): start_timer_thread(): CreateThread() failed, errno 0 15:25:27 (93152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:26:27 (90584): start_timer_thread(): CreateThread() failed, errno 0 15:26:57 (106652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:27:59 (100060): start_timer_thread(): CreateThread() failed, errno 0 15:28:29 (99676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:29:31 (105744): start_timer_thread(): CreateThread() failed, errno 0 15:30:02 (95080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:30:37 (102044): start_timer_thread(): CreateThread() failed, errno 0 15:31:07 (82660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:32:33 (104732): start_timer_thread(): CreateThread() failed, errno 0 15:33:09 (69508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:44 (105872): start_timer_thread(): CreateThread() failed, errno 0 15:34:15 (106152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:50 (101936): start_timer_thread(): CreateThread() failed, errno 0 15:35:20 (103168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:36:21 (108512): start_timer_thread(): CreateThread() failed, errno 0 15:36:51 (102088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:26 (108248): start_timer_thread(): CreateThread() failed, errno 0 15:37:57 (108300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:31 (107844): start_timer_thread(): CreateThread() failed, errno 0 15:39:02 (107664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:39:59 (109192): start_timer_thread(): CreateThread() failed, errno 0 15:40:30 (108696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:41:30 (109552): start_timer_thread(): CreateThread() failed, errno 0 15:42:01 (109396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:43:03 (98184): start_timer_thread(): CreateThread() failed, errno 0 15:43:33 (109300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:44:35 (109304): start_timer_thread(): CreateThread() failed, errno 0 15:45:06 (109196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:45:41 (107660): start_timer_thread(): CreateThread() failed, errno 0 15:46:11 (108636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:13 (109272): start_timer_thread(): CreateThread() failed, errno 0 15:47:44 (109364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:48:18 (109540): start_timer_thread(): CreateThread() failed, errno 0 15:48:49 (109176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:49:24 (108876): start_timer_thread(): CreateThread() failed, errno 0 15:49:55 (109312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:50:29 (109432): start_timer_thread(): CreateThread() failed, errno 0 15:51:00 (104536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:52:01 (109528): start_timer_thread(): CreateThread() failed, errno 0 15:52:32 (108072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:53:07 (109120): start_timer_thread(): CreateThread() failed, errno 0 15:53:37 (107608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:54:12 (103292): start_timer_thread(): CreateThread() failed, errno 0 15:54:43 (108200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:17 (103408): start_timer_thread(): CreateThread() failed, errno 0 15:55:48 (18104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:56:47 (90672): start_timer_thread(): CreateThread() failed, errno 0 15:57:18 (108068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:58:19 (104776): start_timer_thread(): CreateThread() failed, errno 0 15:58:50 (99660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:59:51 (44780): start_timer_thread(): CreateThread() failed, errno 0 16:00:22 (108976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:01:21 (110084): start_timer_thread(): CreateThread() failed, errno 0 16:01:52 (109312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:03:16 (110044): start_timer_thread(): CreateThread() failed, errno 0 16:03:53 (109592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:04:28 (109880): start_timer_thread(): CreateThread() failed, errno 0 16:04:59 (110248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:05:33 (71924): start_timer_thread(): CreateThread() failed, errno 0 16:06:04 (109160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:39 (109324): start_timer_thread(): CreateThread() failed, errno 0 16:07:09 (110272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:07:44 (110600): start_timer_thread(): CreateThread() failed, errno 0 16:08:15 (110536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:08:49 (109592): start_timer_thread(): CreateThread() failed, errno 0 16:09:20 (104752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:09:54 (110908): start_timer_thread(): CreateThread() failed, errno 0 16:10:25 (109964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:11:00 (111052): start_timer_thread(): CreateThread() failed, errno 0 16:11:30 (111144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:05 (111108): start_timer_thread(): CreateThread() failed, errno 0 16:12:36 (110772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:13:00 (110088): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 18:22:21 (111892): start_timer_thread(): CreateThread() failed, errno 0 18:22:48 (44044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:24:20 (115364): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:36:41 (118792): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 00:52:12 (122132): start_timer_thread(): CreateThread() failed, errno 0 05:18:23 (122572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:19:05 (136208): No heartbeat from core client for 30 sec - exiting 05:19:06 (137864): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 05:20:17 (123244): No heartbeat from core client for 30 sec - exiting 05:20:19 (123560): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 05:20:58 (129572): start_timer_thread(): CreateThread() failed, errno 0 05:21:51 (137900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:25 (123632): No heartbeat from core client for 30 sec - exiting 05:22:26 (131416): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 05:23:42 (123744): start_timer_thread(): CreateThread() failed, errno 0 05:23:44 (134880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:51 (135716): start_timer_thread(): CreateThread() failed, errno 0 05:24:22 (138236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:05 (136912): No heartbeat from core client for 30 sec - exiting 05:25:05 (132476): start_timer_thread(): CreateThread() failed, errno 0 05:25:06 (136912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:12 (118984): start_timer_thread(): CreateThread() failed, errno 0 06:51:21 (136448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:02 (137812): No heartbeat from core client for 30 sec - exiting 06:52:03 (130696): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:52:44 (132864): No heartbeat from core client for 30 sec - exiting 06:52:44 (104428): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:53:52 (132564): start_timer_thread(): CreateThread() failed, errno 0 06:54:19 (137828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:01 (135048): No heartbeat from core client for 30 sec - exiting 06:55:01 (136340): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:55:43 (134632): No heartbeat from core client for 30 sec - exiting 06:55:44 (116488): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:56:25 (135600): No heartbeat from core client for 30 sec - exiting 06:56:25 (134992): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:57:07 (132072): No heartbeat from core client for 30 sec - exiting 06:57:08 (132136): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:57:49 (137896): No heartbeat from core client for 30 sec - exiting 06:57:49 (131380): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:58:31 (129656): No heartbeat from core client for 30 sec - exiting 06:58:32 (129884): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 06:59:13 (135600): No heartbeat from core client for 30 sec - exiting 06:59:13 (136636): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 07:00:03 (134812): No heartbeat from core client for 30 sec - exiting 07:00:03 (133072): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 07:00:44 (132564): No heartbeat from core client for 30 sec - exiting 07:00:45 (134812): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 07:01:20 (129816): start_timer_thread(): CreateThread() failed, errno 0 07:02:01 (135664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:02:07 (85684): start_timer_thread(): CreateThread() failed, errno 0 07:02:42 (136560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:05 (87748): start_timer_thread(): CreateThread() failed, errno 0 07:03:29 (136348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:35 (23268): start_timer_thread(): CreateThread() failed, errno 0 07:04:12 (134452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:04:16 (128516): start_timer_thread(): CreateThread() failed, errno 0 07:34:28 (134992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:46 (136832): start_timer_thread(): CreateThread() failed, errno 0 08:01:16 (128528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:19 (136048): start_timer_thread(): CreateThread() failed, errno 0 08:01:56 (86164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:01 (136876): start_timer_thread(): CreateThread() failed, errno 0 11:35:20 (139564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:35:24 (119816): start_timer_thread(): CreateThread() failed, errno 0 11:44:19 (144256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:23 (142016): start_timer_thread(): CreateThread() failed, errno 0 11:45:00 (144720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:04 (144560): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 11:28:53 (135672): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:35:04 (128956): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:36:04 (136612): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:37:19 (136644): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 11:38:19 (137036): start_timer_thread(): CreateThread() failed, errno 0 11:41:00 (131592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:41:13 (135552): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:47:33 (20652): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:48:23 (19460): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:49:04 (136768): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:49:59 (128536): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 11:50:54 (133296): start_timer_thread(): CreateThread() failed, errno 0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Mar 2016 17:00:00	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	259,200	159,942	0.6171
16 Mar 2016 10:07:01	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	233,280	173,778	0.7449
16 Mar 2016 02:59:13	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	207,360	160,846	0.7757
23 Feb 2016 07:42:45	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	181,440	146,550	0.8077
22 Feb 2016 18:41:56	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	155,520	121,163	0.7791
18 Feb 2016 09:20:20	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	129,600	96,346	0.7434
16 Feb 2016 16:48:35	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	103,680	70,347	0.6785
30 Jan 2016 05:14:20	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	77,760	45,634	0.5869
23 Jan 2016 04:58:53	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	51,840	21,162	0.4082
17 Jan 2016 15:51:22	1359488	19133211	hadcm3n_sa1x_198012_480_010221804_0	25,920	25,351	0.9780