Task 19144164

Name	hadcm3n_scfw_198012_480_010221549_1
Workunit	10221549
Created	10 Dec 2015, 10:20:19 UTC
Sent	10 Dec 2015, 10:20:58 UTC
Report deadline	21 Nov 2016, 15:40:58 UTC
Received	20 Dec 2015, 13:59:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1344136
Run time	2 days 21 hours 24 min 21 sec
CPU time	2 days 20 hours 11 min 8 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	3.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.6.9</core_client_version> <![CDATA[ <message> Za - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:35:28 (8020): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 10:08:37 (8080): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 12:59:03 (7356): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 13:10:19 (6932): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:34:25 (5688): start_timer_thread(): CreateThread() failed, errno 0 15:34:54 (7044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:35:46 (7356): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:35:24 (5644): start_timer_thread(): CreateThread() failed, errno 0 16:35:53 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:36:20 (8044): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:04:21 (1236): start_timer_thread(): CreateThread() failed, errno 0 19:04:51 (7296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:15 (708): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 13:13:18 (3936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:43:28 (4728): start_timer_thread(): CreateThread() failed, errno 0 17:43:58 (6360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:16 (6564): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:26:23 (1048): start_timer_thread(): CreateThread() failed, errno 0 20:26:52 (1392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:17 (6140): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:29:42 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:07:59 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:26:28 (8160): start_timer_thread(): CreateThread() failed, errno 0 20:26:58 (1116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:07 (5292): No heartbeat from core client for 30 sec - exiting 20:30:08 (5292): No heartbeat from core client for 30 sec - exiting 20:30:09 (5292): No heartbeat from core client for 30 sec - exiting 20:30:10 (5292): No heartbeat from core client for 30 sec - exiting 20:30:11 (5292): No heartbeat from core client for 30 sec - exiting 20:30:12 (5292): No heartbeat from core client for 30 sec - exiting 20:30:12 (6784): start_timer_thread(): CreateThread() failed, errno 0 20:30:13 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:16 (7400): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:32:45 (4644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:35:33 (6304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:15:01 (9052): start_timer_thread(): CreateThread() failed, errno 0 21:15:39 (8904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:11 (8404): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 21:31:22 (8392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:34 (9164): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:26:41 (3708): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 17:12:51 (7508): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:14:18 (6188): start_timer_thread(): CreateThread() failed, errno 0 21:14:48 (6584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:15:34 (1200): start_timer_thread(): CreateThread() failed, errno 0 21:16:04 (8380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:16:24 (2220): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:14:17 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:48:59 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:29:08 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:19 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:29 (2200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:34:50 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:36:14 (7700): No heartbeat from core client for 30 sec - exiting 17:36:15 (7700): No heartbeat from core client for 30 sec - exiting 17:36:16 (7700): No heartbeat from core client for 30 sec - exiting 17:36:17 (7700): No heartbeat from core client for 30 sec - exiting 17:36:18 (7700): No heartbeat from core client for 30 sec - exiting 17:36:19 (7700): No heartbeat from core client for 30 sec - exiting 17:36:20 (7700): No heartbeat from core client for 30 sec - exiting 17:36:21 (7700): No heartbeat from core client for 30 sec - exiting 17:36:22 (7700): No heartbeat from core client for 30 sec - exiting 17:36:23 (7700): No heartbeat from core client for 30 sec - exiting 17:36:24 (7700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:39:28 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:47:38 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:08:45 (8200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:06:45 (1416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:18:19 (7832): No heartbeat from core client for 30 sec - exiting 11:18:19 (3316): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - No 'heartbeat' from BOINC... 11:19:59 (7404): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 11:23:06 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:08 (5640): start_timer_thread(): CreateThread() failed, errno 0 11:23:38 (8040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:43 (8028): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:01:15 (8472): start_timer_thread(): CreateThread() failed, errno 0 14:01:25 (6288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:02:21 (2928): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... 14:10:48 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:11:14 (8212): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:21:35 (7208): start_timer_thread(): CreateThread() failed, errno 0 14:22:07 (8100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:23:02 (5672): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 12:04:09 (956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:42:43 (7824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2015 14:33:13	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	207,360	247,962	1.1958
17 Dec 2015 20:37:46	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	181,440	216,037	1.1907
16 Dec 2015 07:18:23	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	155,520	185,521	1.1929
15 Dec 2015 17:41:10	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	129,600	142,831	1.1021
15 Dec 2015 06:06:51	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	103,680	101,651	0.9804
14 Dec 2015 18:38:55	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	77,760	60,533	0.7785
12 Dec 2015 18:57:55	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	51,840	40,568	0.7826
11 Dec 2015 13:27:47	1344136	19144164	hadcm3n_scfw_198012_480_010221549_1	25,920	24,070	0.9286