Task 15196069

Name	hadam3p_eu_wq1j_1966_1_006877103_2
Workunit	7080419
Created	28 Aug 2012, 14:27:24 UTC
Sent	28 Aug 2012, 14:33:51 UTC
Report deadline	10 Aug 2013, 19:53:51 UTC
Received	30 Sep 2012, 18:36:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1168410
Run time	5 days 1 hours 51 min 22 sec
CPU time	4 days 5 hours 43 min 22 sec
Validate state	Invalid
Credit	2,187.67
Device peak FLOPS	2.23 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <stderr_txt> 19:45:03 (3968): start_timer_thread(): CreateThread() failed, errno 0 15:47:15 (6884): No heartbeat from core client for 30 sec - exiting 15:47:16 (6884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:47:57 (2496): start_timer_thread(): CreateThread() failed, errno 0 19:25:12 (6128): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7368, iMonCtr=2 Model crash detected, will try to restart... 20:56:23 (7972): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1052, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 10:24:09 (7132): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5100, selfPID=7136, iMonCtr=1 Model crash detected, will try to restart... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 17:48:00 (5028): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 17:48:00 (2140): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2140, selfPID=3780, iMonCtr=1 Model crash detected, will try to restart... 18:56:21 (7284): start_timer_thread(): CreateThread() failed, errno 0 18:56:22 (7276): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7276, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5924, iMonCtr=2 Model crash detected, will try to restart... 18:06:11 (2128): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:06:12 (6980): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2128, selfPID=6492, iMonCtr=1 Model crash detected, will try to restart... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=708, selfPID=600, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1784, selfPID=6228, iMonCtr=1 Model crash detected, will try to restart... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 20:53:03 (7400): start_timer_thread(): CreateThread() failed, errno 0 10:09:10 (6328): start_timer_thread(): CreateThread() failed, errno 0 10:09:10 (4800): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5924, iMonCtr=2 Model crash detected, will try to restart... 16:38:41 (2504): start_timer_thread(): CreateThread() failed, errno 0 18:06:08 (7296): start_timer_thread(): CreateThread() failed, errno 0 18:06:09 (7304): start_timer_thread(): CreateThread() failed, errno 0 19:41:28 (3248): start_timer_thread(): CreateThread() failed, errno 0 19:41:28 (7372): start_timer_thread(): CreateThread() failed, errno 0 10:23:30 (6896): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2480, iMonCtr=2 Model crash detected, will try to restart... 18:56:59 (6856): start_timer_thread(): CreateThread() failed, errno 0 18:57:00 (3788): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3944, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6856, iMonCtr=2 di2:49agnos81ics_instart_timdlr_thread(ti: CrmonitThr(): Creating hEerceptio nMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8140, selfPID=6824, iMonCtr=1 Model crash detected, will try to restart... 13:55:22 (2112): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8008, iMonCtr=2 19:25:00 (7736): start_timer_thread(): CreateThread() failed, errno 0 19:25:01 (6524): start_timer_thread(): CreateThread() failed, errno 0 11:33:09 (6364):7812)rt_timer_timer_thread(): Created(hread() failed, err no 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6364, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6656, iMonCtr=2 Model crash detected, will try to restart... 15:13:41 (7424): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 188:46:460 (5737564): start_termeth_teare()d( C: eareeTterehrd(ad fa() fai eedrn e 0rn o 0 CPDN Monitor - Quit request from BOINC... 21:29:32 (5184): start_timer_thread(): CreateThread() failed, errno 0 10:30:15 (7936): start_timer_thread(): CreateThread() failed, errno 0 10:30:15 (7944): start_timer_thread(): CreateThread() failed, errno 0 19:08:20 (8056): start_timer_thread(): CreateThread() failed, errno 0 19:08:28 (8064): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4300, selfPID=6160, iMonCtr=1 Model crash detected, will try to restart... 09:04:45 (7044): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 20:08:09 (7484): start_timer_thread(): CreateThread() failed, errno 0 20:08:10 (1516): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1516, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=968, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6416, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6052, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:29:18 (6540): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 18:29:19 (6548): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6540, selfPID=6540, iMonCtr=2 18:34:07 (4652): start_timer_thread(): CreateThread() failed, errno 0 11:52:17 (7924(7: sta: start_t_thread(): CreateThrtead() failed, errno 0 no 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7924, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4924, iMonCtr=2 20:43:37 (7372): start_timer_thread(): CreateThread() failed, errno 0 20:43:37 (7364): start_timer_thread(): CreateThread() failed, errno 0 diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3416, selfPID=3424, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:07:18 (4084): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7480, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 10:10:31 (7108): start_timer_thread(): CreateThread() failed, errno 0 11:53:58 (6904): start_timer_thread(): CreateThread() failed, errno 0 11:53:59 (6916): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1624, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6904, iMonCtr=2 Leaving CPDN_Main::Monitor... 16:33:07 (2104): start_timer_thread(): CreateThread() failed, errno 0 16:33:07 (4452): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4452, selfPID=7380, iMonCtr=1 Model crash detected, will try to restart... diagnostics_init_unhandled_exception_monitor(): Creating hExceptionMonitorThread failed, errno 12 WARNING: BOINC Windows Runtime Debugger has been disabled. 11:32:38 (4312): start_timer_thread(): CreateThread() failed, errno 0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6916, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 17:26:17 (5748): start_timer_thread(): CreateThread() failed, errno 0 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4420, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7724, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadam3p_eu_wq1j_1966_1_006877103/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadam3p_eu_wq1j_1966_1_006877103/dataout/region_restart.day after 11 attempts forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_eu_wq1j_1966_1_006877103\tmp\xaakm.namelists Image PC Routine Line Source hadam3p_eu_um_6.0 011BA39A Unknown Unknown Unknown hadam3p_eu_um_6.0 01162CD0 Unknown Unknown Unknown hadam3p_eu_um_6.0 01161E9A Unknown Unknown Unknown hadam3p_eu_um_6.0 01142819 Unknown Unknown Unknown hadam3p_eu_um_6.0 01042287 Unknown Unknown Unknown hadam3p_eu_um_6.0 010DE7B2 Unknown Unknown Unknown hadam3p_eu_um_6.0 010DF2DA Unknown Unknown Unknown hadam3p_eu_um_6.0 00E59BD2 Unknown Unknown Unknown hadam3p_eu_um_6.0 0119E638 Unknown Unknown Unknown kernel32.dll 77A2D309 Unknown Unknown Unknown ntdll.dll 77CE1603 Unknown Unknown Unknown ntdll.dll 77CE15D6 Unknown Unknown Unknown forrtl: severe (24): end-of-file during read, unit 9, file C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_eu_wq1j_1966_1_006877103\tmp\xaakg.namelists Image PC Routine Line Source hadrm3p_eu_um_6.0 00E5C52A Unknown Unknown Unknown hadrm3p_eu_um_6.0 00E04460 Unknown Unknown Unknown hadrm3p_eu_um_6.0 00E0362A Unknown Unknown Unknown hadrm3p_eu_um_6.0 00DE2469 Unknown Unknown Unknown hadrm3p_eu_um_6.0 00CE66EB Unknown Unknown Unknown hadrm3p_eu_um_6.0 00D82AE2 Unknown Unknown Unknown hadrm3p_eu_um_6.0 00D835AF Unknown Unknown Unknown hadrm3p_eu_um_6.0 00B29860 Unknown Unknown Unknown hadrm3p_eu_um_6.0 00E40893 Unknown Unknown Unknown kernel32.dll 77A2D309 Unknown Unknown Unknown ntdll.dll 77CE1603 Unknown Unknown Unknown ntdll.dll 77CE15D6 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7668, selfPID=7156, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_eu_wq1j_1966_1_006877103_2_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2012 13:17:23	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	126,816	360,123	2.8397
28 Sep 2012 20:01:51	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	115,296	324,114	2.8111
27 Sep 2012 20:22:37	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	103,776	291,025	2.8044
26 Sep 2012 16:47:14	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	92,261	258,319	2.7999
26 Sep 2012 14:50:18	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	92,256	257,911	2.7956
25 Sep 2012 13:23:59	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	80,736	226,223	2.8020
24 Sep 2012 12:05:45	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	69,216	192,545	2.7818
22 Sep 2012 17:37:23	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	57,696	160,840	2.7877
21 Sep 2012 09:32:59	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	46,176	129,581	2.8062
19 Sep 2012 18:04:45	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	34,656	97,601	2.8163
18 Sep 2012 18:21:34	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	23,136	64,823	2.8018
17 Sep 2012 10:50:42	1168410	15196069	hadam3p_eu_wq1j_1966_1_006877103_2	11,616	33,064	2.8464