Task 16649420

Name	hadam3pm2_e8ms_1991_10_008714231_1
Workunit	8860705
Created	27 May 2014, 21:02:57 UTC
Sent	27 May 2014, 21:03:28 UTC
Report deadline	10 May 2015, 2:23:28 UTC
Received	14 Jun 2014, 8:50:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1399008
Run time	10 days 23 hours 4 min 22 sec
CPU time	8 days 1 hours 18 min 18 sec
Validate state	Invalid
Credit	8,900.08
Device peak FLOPS	2.57 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> oa.pc\|1dec.nc oa.pc\|2jan.nc oa.pc\|2feb.nc oa.pc\|2mar.nc oa.pc\|2apr.nc oa.pc\|2may.nc oa.pc\|2jun.nc oa.pc\|2jul.nc oa.pc\|2aug.nc oa.pc\|2sep.nc oa.pc\|2oct.nc oa.pc\|2nov.nc oa.pe\|1dec.nc oa.pe\|2jan.nc oa.pe\|2feb.nc oa.pe\|2mar.nc oa.pe\|2apr.nc oa.pe\|2may.nc oa.pe\|2jun.nc oa.pe\|2jul.nc oa.pe\|2aug.nc oa.pe\|2sep.nc oa.pe\|2oct.nc oa.pe\|2nov.nc oa.pc\|2dec.nc oa.pc\|3jan.nc oa.pc\|3feb.nc oa.pc\|3mar.nc oa.pc\|3apr.nc oa.pc\|3may.nc oa.pc\|3jun.nc oa.pc\|3jul.nc oa.pc\|3aug.nc oa.pc\|3sep.nc oa.pc\|3oct.nc oa.pc\|3nov.nc oa.pe\|2dec.nc oa.pe\|3jan.nc oa.pe\|3feb.nc oa.pe\|3mar.nc oa.pe\|3apr.nc oa.pe\|3may.nc oa.pe\|3jun.nc oa.pe\|3jul.nc oa.pe\|3aug.nc oa.pe\|3sep.nc oa.pe\|3oct.nc oa.pe\|3nov.nc oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc 04:21:19 (1357): No heartbeat from client for 30 sec - exiting 04:21:34 (1357): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... oa.pc\|5dec.nc oa.pc\|6jan.nc oa.pc\|6feb.nc oa.pc\|6mar.nc oa.pc\|6apr.nc oa.pc\|6may.nc oa.pc\|6jun.nc oa.pc\|6jul.nc oa.pc\|6aug.nc oa.pc\|6sep.nc oa.pc\|6oct.nc oa.pc\|6nov.nc oa.pe\|5dec.nc oa.pe\|6jan.nc oa.pe\|6feb.nc oa.pe\|6mar.nc oa.pe\|6apr.nc oa.pe\|6may.nc oa.pe\|6jun.nc oa.pe\|6jul.nc oa.pe\|6aug.nc oa.pe\|6sep.nc oa.pe\|6oct.nc oa.pe\|6nov.nc oa.pc\|6dec.nc oa.pc\|7jan.nc oa.pc\|7feb.nc oa.pc\|7mar.nc oa.pc\|7apr.nc oa.pc\|7may.nc oa.pc\|7jun.nc oa.pc\|7jul.nc oa.pc\|7aug.nc oa.pc\|7sep.nc oa.pc\|7oct.nc oa.pc\|7nov.nc oa.pe\|6dec.nc oa.pe\|7jan.nc oa.pe\|7feb.nc oa.pe\|7mar.nc oa.pe\|7apr.nc oa.pe\|7may.nc oa.pe\|7jun.nc oa.pe\|7jul.nc oa.pe\|7aug.nc oa.pe\|7sep.nc oa.pe\|7oct.nc oa.pe\|7nov.nc 00:09:38 (22929): No heartbeat from client for 30 sec - exiting 00:09:39 (22929): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:50 (7446): No heartbeat from client for 30 sec - exiting 00:10:50 (7446): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:29 (7492): No heartbeat from client for 30 sec - exiting 00:12:29 (7492): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:16:13 (7543): No heartbeat from client for 30 sec - exiting 00:16:13 (7543): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:21:59 (7635): No heartbeat from client for 30 sec - exiting 00:21:59 (7635): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:23:16 (7783): No heartbeat from client for 30 sec - exiting 00:23:16 (7783): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:24:38 (7835): No heartbeat from client for 30 sec - exiting 00:24:38 (7835): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:42 (7882): No heartbeat from client for 30 sec - exiting 06:48:42 (7882): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:49:25 (15625): No heartbeat from client for 30 sec - exiting 06:49:25 (15625): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:35 (15656): No heartbeat from client for 30 sec - exiting 06:50:35 (15656): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:52:15 (15714): No heartbeat from client for 30 sec - exiting 06:52:15 (15714): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:11 (15754): No heartbeat from client for 30 sec - exiting 06:53:11 (15754): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:18 (15793): No heartbeat from client for 30 sec - exiting 06:55:18 (15793): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:56:28 (15852): No heartbeat from client for 30 sec - exiting 06:56:28 (15852): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:14 (15893): No heartbeat from client for 30 sec - exiting 06:57:14 (15893): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:16 (15929): No heartbeat from client for 30 sec - exiting 06:59:16 (15929): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:01:28 (15988): No heartbeat from client for 30 sec - exiting 07:01:28 (15988): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:04:10 (16048): No heartbeat from client for 30 sec - exiting 07:04:10 (16048): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:30 (16124): No heartbeat from client for 30 sec - exiting 07:05:30 (16124): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:07 (16183): No heartbeat from client for 30 sec - exiting 07:08:07 (16183): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:10:40 (16239): No heartbeat from client for 30 sec - exiting 07:10:40 (16239): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:12:32 (16315): No heartbeat from client for 30 sec - exiting 07:12:32 (16315): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:18 (16372): No heartbeat from client for 30 sec - exiting 07:14:18 (16372): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:30 (16426): No heartbeat from client for 30 sec - exiting 07:16:30 (16426): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:18:22 (16488): No heartbeat from client for 30 sec - exiting 07:18:22 (16488): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:13 (16565): No heartbeat from client for 30 sec - exiting 07:20:13 (16565): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:45 (16620): No heartbeat from client for 30 sec - exiting 07:22:45 (16620): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:24:41 (16686): No heartbeat from client for 30 sec - exiting 07:24:41 (16686): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:13 (16748): No heartbeat from client for 30 sec - exiting 07:27:13 (16748): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:14 (16815): No heartbeat from client for 30 sec - exiting 07:28:14 (16815): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:51 (16856): No heartbeat from client for 30 sec - exiting 07:29:51 (16856): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:32:18 (16907): No heartbeat from client for 30 sec - exiting 07:32:18 (16907): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:55 (16978): No heartbeat from client for 30 sec - exiting 07:34:55 (16978): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:27 (17089): No heartbeat from client for 30 sec - exiting 07:37:27 (17089): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:58 (17156): No heartbeat from client for 30 sec - exiting 07:39:58 (17156): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:42:26 (17254): No heartbeat from client for 30 sec - exiting 07:42:26 (17254): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:45:01 (17324): No heartbeat from client for 30 sec - exiting 07:45:01 (17324): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:47:43 (17399): No heartbeat from client for 30 sec - exiting 07:47:43 (17399): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:49:55 (17469): No heartbeat from client for 30 sec - exiting 07:49:55 (17469): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:57 (17537): No heartbeat from client for 30 sec - exiting 07:51:57 (17537): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:53:53 (17598): No heartbeat from client for 30 sec - exiting 07:53:53 (17598): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:05 (17654): No heartbeat from client for 30 sec - exiting 07:56:05 (17654): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:27 (17716): No heartbeat from client for 30 sec - exiting 07:58:27 (17716): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:38 (17782): No heartbeat from client for 30 sec - exiting 08:00:38 (17782): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:20 (17842): No heartbeat from client for 30 sec - exiting 08:03:20 (17842): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:42 (17919): No heartbeat from client for 30 sec - exiting 08:05:42 (17919): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:00 (17982): No heartbeat from client for 30 sec - exiting 08:08:00 (17982): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:10:00 (18043): No heartbeat from client for 30 sec - exiting 08:10:00 (18043): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:06 (18108): No heartbeat from client for 30 sec - exiting 08:12:06 (18108): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:38 (18171): No heartbeat from client for 30 sec - exiting 08:14:38 (18171): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:30 (18241): No heartbeat from client for 30 sec - exiting 08:16:30 (18241): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:42 (18296): No heartbeat from client for 30 sec - exiting 08:18:42 (18296): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:21:15 (18378): No heartbeat from client for 30 sec - exiting 08:21:15 (18378): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:27 (18444): No heartbeat from client for 30 sec - exiting 08:23:27 (18444): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:24:08 (18511): No heartbeat from client for 30 sec - exiting 08:24:08 (18511): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:20 (18547): No heartbeat from client for 30 sec - exiting 08:26:20 (18547): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:51 (18602): No heartbeat from client for 30 sec - exiting 08:28:51 (18602): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:31:18 (18672): No heartbeat from client for 30 sec - exiting 08:31:18 (18672): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:20 (18736): No heartbeat from client for 30 sec - exiting 08:33:20 (18736): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:32 (18794): No heartbeat from client for 30 sec - exiting 08:35:32 (18794): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:37:33 (18856): No heartbeat from client for 30 sec - exiting 08:37:33 (18856): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:39:20 (18919): No heartbeat from client for 30 sec - exiting 08:39:20 (18919): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:27 (18973): No heartbeat from client for 30 sec - exiting 08:41:27 (18973): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:43:18 (19033): No heartbeat from client for 30 sec - exiting 08:43:18 (19033): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:55 (19097): No heartbeat from client for 30 sec - exiting 08:45:55 (19097): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:43 (19166): No heartbeat from client for 30 sec - exiting 08:47:43 (19166): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:24 (19221): No heartbeat from client for 30 sec - exiting 08:50:24 (19221): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... oa.pc\|8dec.nc oa.pc\|9jan.nc oa.pc\|9feb.nc oa.pc\|9mar.nc oa.pc\|9apr.nc oa.pc\|9may.nc oa.pc\|9jun.nc oa.pc\|9jul.nc oa.pc\|9aug.nc oa.pc\|9sep.nc oa.pc\|9oct.nc oa.pc\|9nov.nc oa.pe\|8dec.nc oa.pe\|9jan.nc oa.pe\|9feb.nc oa.pe\|9mar.nc oa.pe\|9apr.nc oa.pe\|9may.nc oa.pe\|9jun.nc oa.pe\|9jul.nc oa.pe\|9aug.nc oa.pe\|9sep.nc oa.pe\|9oct.nc oa.pe\|9nov.nc oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Sorry, too many model crashes! :-( 04:08:40 (3009): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jun 2014 03:08:37	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	308,228	914,789	2.9679
14 Jun 2014 00:58:25	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	305,348	906,502	2.9688
13 Jun 2014 22:38:11	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	302,468	898,313	2.9699
13 Jun 2014 20:17:58	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	299,588	890,061	2.9710
13 Jun 2014 17:57:41	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	296,708	881,808	2.9720
13 Jun 2014 15:42:29	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	293,828	873,538	2.9730
13 Jun 2014 13:27:16	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	290,948	865,329	2.9742
13 Jun 2014 11:17:05	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	288,068	857,072	2.9752
13 Jun 2014 09:16:54	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	285,188	848,846	2.9764
13 Jun 2014 07:16:44	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	282,308	840,657	2.9778
13 Jun 2014 04:06:28	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	279,428	832,077	2.9778
12 Jun 2014 23:36:06	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	273,668	814,872	2.9776
12 Jun 2014 21:35:55	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	270,788	806,192	2.9772
12 Jun 2014 18:20:38	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	267,908	797,457	2.9766
12 Jun 2014 15:55:24	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	265,028	788,724	2.9760
12 Jun 2014 13:29:51	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	262,148	780,016	2.9755
12 Jun 2014 10:59:37	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	259,268	771,314	2.9750
12 Jun 2014 09:14:26	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	256,388	762,618	2.9745
12 Jun 2014 06:29:10	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	253,508	753,994	2.9742
12 Jun 2014 03:38:55	1229996	16649420	hadam3pm2_e8ms_1991_10_008714231_1	250,628	745,373	2.9740