Task 19239647

Name	hadam3p_anz_n9rl_201412_12_315_010288199_0
Workunit	10288199
Created	29 Jan 2016, 13:13:06 UTC
Sent	1 Feb 2016, 12:59:39 UTC
Report deadline	13 Jan 2017, 18:19:39 UTC
Received	25 Feb 2016, 3:05:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID	1257567
Run time	7 days 1 hours 7 min 9 sec
CPU time	6 days 12 hours 4 min 21 sec
Validate state	Invalid
Credit	4,484.28
Device peak FLOPS	2.91 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> finish file present too long </message> <stderr_txt> 10:18:49 (15964): start_timer_thread(): CreateThread() failed, errno 0 10:18:50 (8172): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 16:29:04 (10208): start_timer_thread(): CreateThread() failed, errno 0 16:29:06 (7960): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:24:42 (14692): start_timer_thread(): CreateThread() failed, errno 0 22:24:44 (10236): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:46:39 (8044): start_timer_thread(): CreateThread() failed, errno 0 01:46:41 (8152): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:00:47 (10180): start_timer_thread(): CreateThread() failed, errno 0 15:00:49 (10512): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:53:19 (10820): start_timer_thread(): CreateThread() failed, errno 0 08:53:21 (10872): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 00:48:15 (10292): start_timer_thread(): CreateThread() failed, errno 0 00:48:17 (5624): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:46:21 (6672): start_timer_thread(): CreateThread() failed, errno 0 05:46:23 (2232): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 06:37:51 (7808): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:10:46 (7612): start_timer_thread(): CreateThread() failed, errno 0 13:10:48 (5344): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:22:05 (12128): start_timer_thread(): CreateThread() failed, errno 0 20:22:07 (3744): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:46:43 (11952): start_timer_thread(): CreateThread() failed, errno 0 21:46:45 (10984): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:49:51 (12000): start_timer_thread(): CreateThread() failed, errno 0 15:49:53 (4588): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 18:18:39 (2296): start_timer_thread(): CreateThread() failed, errno 0 18:18:41 (11932): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:27:35 (9532): start_timer_thread(): CreateThread() failed, errno 0 18:27:38 (2852): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:08:13 (4000): start_timer_thread(): CreateThread() failed, errno 0 09:08:15 (7276): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:22:16 (1144): start_timer_thread(): CreateThread() failed, errno 0 16:22:18 (11144): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 01:03:24 (4208): start_timer_thread(): CreateThread() failed, errno 0 01:03:26 (10252): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:42 (10532): start_timer_thread(): CreateThread() failed, errno 0 00:19:17 (6512): start_timer_thread(): CreateThread() failed, errno 0 00:19:18 (7128): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 22:41:50 (6004): start_timer_thread(): CreateThread() failed, errno 0 22:41:53 (11096): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:52:32 (1204): start_timer_thread(): CreateThread() failed, errno 0 13:52:34 (5756): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 19:16:39 (5596): start_timer_thread(): CreateThread() failed, errno 0 19:16:40 (6664): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:41:00 (12312): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 08:38:25 (5828): start_timer_thread(): CreateThread() failed, errno 0 08:38:26 (6620): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 08:56:39 (15136): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:17:22 (5684): start_timer_thread(): CreateThread() failed, errno 0 18:17:24 (16924): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 23:20:22 (360): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 23:20:23 (15164): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 15:39:30 (9728): start_timer_thread(): CreateThread() failed, errno 0 15:39:32 (9756): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:28:21 (3824): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:16:44 (12332): start_timer_thread(): CreateThread() failed, errno 0 16:16:46 (1660): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:07 (9344): start_timer_thread(): CreateThread() failed, errno 0 19:55:09 (8992): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:35:46 (9180): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 08:53:06 (6844): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:45:33 (2728): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:16:42 (8092): start_timer_thread(): CreateThread() failed, errno 0 13:16:44 (3096): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 21:19:35 (2096): start_timer_thread(): CreateThread() failed, errno 0 21:19:37 (12184): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:29:50 (11644): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11644, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9972, selfPID=11712, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 23:53:11 (11712): No heartbeat from core client for 30 sec - exiting 23:53:12 (11712): No heartbeat from core client for 30 sec - exiting 23:53:13 (11712): No heartbeat from core client for 30 sec - exiting 23:53:14 (11712): No heartbeat from core client for 30 sec - exiting Called boinc_finish 23:53:15 (11712): No heartbeat from core client for 30 sec - exiting 23:53:16 (11712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Abort request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Feb 2016 02:08:19	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	103,979	517,368	4.9757
23 Feb 2016 00:49:13	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	92,459	458,332	4.9571
21 Feb 2016 08:10:17	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	80,939	407,398	5.0334
18 Feb 2016 13:41:10	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	69,419	347,904	5.0117
17 Feb 2016 20:06:03	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	57,899	286,302	4.9449
16 Feb 2016 15:38:06	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	46,379	224,770	4.8464
08 Feb 2016 13:11:44	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	34,859	165,441	4.7460
04 Feb 2016 10:48:21	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	23,339	104,383	4.4725
02 Feb 2016 03:51:30	1257567	19239647	hadam3p_anz_n9rl_201412_12_315_010288199_0	11,819	45,460	3.8463