Task 12770843

Name	hadcm3n_o02a_1900_40_007195413_2
Workunit	7393693
Created	3 Apr 2011, 13:18:25 UTC
Sent	3 Apr 2011, 13:26:11 UTC
Report deadline	3 Jul 2011, 20:53:22 UTC
Received	7 Apr 2011, 18:34:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1118054
Run time	2 days 10 hours 11 min 52 sec
CPU time	1 days 16 hours 10 min 1 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:57:01 (13971): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:25:00 (3985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:01 (3985): No heartbeat from core client for 30 sec - exiting 15:25:02 (3985): No heartbeat from core client for 30 sec - exiting 15:25:03 (3985): No heartbeat from core client for 30 sec - exiting 15:25:04 (3985): No heartbeat from core client for 30 sec - exiting 15:25:05 (3985): No heartbeat from core client for 30 sec - exiting 15:25:06 (3985): No heartbeat from core client for 30 sec - exiting 15:25:07 (3985): No heartbeat from core client for 30 sec - exiting 15:25:08 (3985): No heartbeat from core client for 30 sec - exiting 15:25:09 (3985): No heartbeat from core client for 30 sec - exiting 15:25:10 (3985): No heartbeat from core client for 30 sec - exiting 15:25:11 (3985): No heartbeat from core client for 30 sec - exiting 15:25:12 (3985): No heartbeat from core client for 30 sec - exiting 15:25:13 (3985): No heartbeat from core client for 30 sec - exiting 15:25:14 (3985): No heartbeat from core client for 30 sec - exiting 15:25:15 (3985): No heartbeat from core client for 30 sec - exiting 15:25:16 (3985): No heartbeat from core client for 30 sec - exiting 15:25:17 (3985): No heartbeat from core client for 30 sec - exiting 15:25:18 (3985): No heartbeat from core client for 30 sec - exiting 15:25:19 (3985): No heartbeat from core client for 30 sec - exiting 15:25:20 (3985): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 17:51:14 (18311): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:03 (19828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:04 (19828): No heartbeat from core client for 30 sec - exiting 17:54:05 (19828): No heartbeat from core client for 30 sec - exiting 17:54:06 (19828): No heartbeat from core client for 30 sec - exiting 17:54:07 (19828): No heartbeat from core client for 30 sec - exiting 17:54:08 (19828): No heartbeat from core client for 30 sec - exiting 17:54:09 (19828): No heartbeat from core client for 30 sec - exiting 17:54:10 (19828): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 18:55:18 (19867): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:19 (19867): No heartbeat from core client for 30 sec - exiting 18:55:20 (19867): No heartbeat from core client for 30 sec - exiting 18:55:21 (19867): No heartbeat from core client for 30 sec - exiting 18:55:22 (19867): No heartbeat from core client for 30 sec - exiting 18:55:23 (19867): No heartbeat from core client for 30 sec - exiting 18:55:24 (19867): No heartbeat from core client for 30 sec - exiting 18:55:25 (19867): No heartbeat from core client for 30 sec - exiting 18:55:26 (19867): No heartbeat from core client for 30 sec - exiting 18:55:27 (19867): No heartbeat from core client for 30 sec - exiting 18:55:28 (19867): No heartbeat from core client for 30 sec - exiting 18:55:29 (19867): No heartbeat from core client for 30 sec - exiting 18:55:30 (19867): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 20:07:30 (21997): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:37:52 (24760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:47:17 (25771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:22:38 (29614): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:22:49 (29614): No heartbeat from core client for 30 sec - exiting 22:22:50 (29614): No heartbeat from core client for 30 sec - exiting 22:22:51 (29614): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:09:20 (29661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:21 (29661): No heartbeat from core client for 30 sec - exiting 07:09:22 (29661): No heartbeat from core client for 30 sec - exiting 07:09:23 (29661): No heartbeat from core client for 30 sec - exiting 07:09:24 (29661): No heartbeat from core client for 30 sec - exiting 07:09:25 (29661): No heartbeat from core client for 30 sec - exiting 07:09:26 (29661): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:57:01 (17313): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:02 (17313): No heartbeat from core client for 30 sec - exiting 07:57:03 (17313): No heartbeat from core client for 30 sec - exiting 07:57:04 (17313): No heartbeat from core client for 30 sec - exiting 07:57:05 (17313): No heartbeat from core client for 30 sec - exiting 07:57:06 (17313): No heartbeat from core client for 30 sec - exiting 07:57:07 (17313): No heartbeat from core client for 30 sec - exiting 07:57:08 (17313): No heartbeat from core client for 30 sec - exiting 07:57:09 (17313): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 10:32:13 (24298): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:32:14 (24298): No heartbeat from core client for 30 sec - exiting 10:32:15 (24298): No heartbeat from core client for 30 sec - exiting 10:32:16 (24298): No heartbeat from core client for 30 sec - exiting 10:32:17 (24298): No heartbeat from core client for 30 sec - exiting 10:32:18 (24298): No heartbeat from core client for 30 sec - exiting 10:32:19 (24298): No heartbeat from core client for 30 sec - exiting 15:53:52 (24726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:49 (4476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:50 (4476): No heartbeat from core client for 30 sec - exiting 16:06:51 (4476): No heartbeat from core client for 30 sec - exiting 16:06:52 (4476): No heartbeat from core client for 30 sec - exiting 16:06:53 (4476): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 16:34:36 (5145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:34:37 (5145): No heartbeat from core client for 30 sec - exiting 16:34:38 (5145): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... 20:24:29 (14394): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:30 (14394): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 20:28:11 (14516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:12 (14516): No heartbeat from core client for 30 sec - exiting 20:28:13 (14516): No heartbeat from core client for 30 sec - exiting 20:28:14 (14516): No heartbeat from core client for 30 sec - exiting 20:28:15 (14516): No heartbeat from core client for 30 sec - exiting 20:28:16 (14516): No heartbeat from core client for 30 sec - exiting 20:28:17 (14516): No heartbeat from core client for 30 sec - exiting 20:28:18 (14516): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Apr 2011 07:06:02	1118054	12770843	hadcm3n_o02a_1900_40_007195413_2	77,760	124,351	1.5992
05 Apr 2011 12:44:56	1118054	12770843	hadcm3n_o02a_1900_40_007195413_2	51,840	83,629	1.6132
04 Apr 2011 12:34:45	1118054	12770843	hadcm3n_o02a_1900_40_007195413_2	25,920	41,823	1.6135