Task 12992382

Name	hadcm3n_s72i_1940_40_007300749_1
Workunit	7498173
Created	21 Jun 2011, 0:47:32 UTC
Sent	21 Jun 2011, 0:47:50 UTC
Report deadline	20 Sep 2011, 8:15:01 UTC
Received	28 Jun 2011, 2:08:27 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1006135
Run time	5 days 23 hours 56 min 27 sec
CPU time	5 days 9 hours 53 min 48 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:28:07 (37703): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:14 (37703): No heartbeat from core client for 30 sec - exiting 08:28:15 (37703): No heartbeat from core client for 30 sec - exiting 08:28:16 (37703): No heartbeat from core client for 30 sec - exiting 08:28:17 (37703): No heartbeat from core client for 30 sec - exiting 08:28:18 (37703): No heartbeat from core client for 30 sec - exiting 08:28:19 (37703): No heartbeat from core client for 30 sec - exiting 08:28:20 (37703): No heartbeat from core client for 30 sec - exiting 08:28:21 (37703): No heartbeat from core client for 30 sec - exiting 08:38:18 (37843): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:38:20 (37843): No heartbeat from core client for 30 sec - exiting 08:38:21 (37843): No heartbeat from core client for 30 sec - exiting 08:38:22 (37843): No heartbeat from core client for 30 sec - exiting 08:55:41 (37920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:41 (38049): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:45 (38049): No heartbeat from core client for 30 sec - exiting 09:43:46 (38049): No heartbeat from core client for 30 sec - exiting 10:00:23 (38385): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:33 (38511): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:07:35 (38511): No heartbeat from core client for 30 sec - exiting 10:07:36 (38511): No heartbeat from core client for 30 sec - exiting 10:17:24 (38578): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:29 (38578): No heartbeat from core client for 30 sec - exiting 10:34:43 (38661): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:34:46 (38661): No heartbeat from core client for 30 sec - exiting 10:34:47 (38661): No heartbeat from core client for 30 sec - exiting 10:34:48 (38661): No heartbeat from core client for 30 sec - exiting 10:34:49 (38661): No heartbeat from core client for 30 sec - exiting 10:51:29 (38783): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:42 (38906): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:19 (38979): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:06 (39149): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:22 (39205): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:45:24 (39205): No heartbeat from core client for 30 sec - exiting 11:53:16 (39336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:18 (39336): No heartbeat from core client for 30 sec - exiting 11:53:21 (39336): No heartbeat from core client for 30 sec - exiting 11:53:22 (39336): No heartbeat from core client for 30 sec - exiting 11:53:23 (39336): No heartbeat from core client for 30 sec - exiting 11:53:24 (39336): No heartbeat from core client for 30 sec - exiting 11:53:25 (39336): No heartbeat from core client for 30 sec - exiting 11:53:26 (39336): No heartbeat from core client for 30 sec - exiting 11:53:27 (39336): No heartbeat from core client for 30 sec - exiting 11:53:28 (39336): No heartbeat from core client for 30 sec - exiting 11:53:29 (39336): No heartbeat from core client for 30 sec - exiting 11:53:30 (39336): No heartbeat from core client for 30 sec - exiting 11:53:31 (39336): No heartbeat from core client for 30 sec - exiting 11:53:32 (39336): No heartbeat from core client for 30 sec - exiting 11:53:33 (39336): No heartbeat from core client for 30 sec - exiting 11:53:34 (39336): No heartbeat from core client for 30 sec - exiting 12:05:58 (39408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:36 (39505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:39 (39505): No heartbeat from core client for 30 sec - exiting 12:35:40 (39505): No heartbeat from core client for 30 sec - exiting 12:35:41 (39505): No heartbeat from core client for 30 sec - exiting 12:48:54 (39720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:09:59 (39817): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:10:01 (39817): No heartbeat from core client for 30 sec - exiting 13:10:02 (39817): No heartbeat from core client for 30 sec - exiting 13:16:53 (39989): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:16:54 (39989): No heartbeat from core client for 30 sec - exiting 13:16:55 (39989): No heartbeat from core client for 30 sec - exiting 13:16:56 (39989): No heartbeat from core client for 30 sec - exiting 13:16:57 (39989): No heartbeat from core client for 30 sec - exiting 13:29:22 (40046): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:19 (40145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:12:02 (40268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:22:12 (40453): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:05 (40536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:07 (40536): No heartbeat from core client for 30 sec - exiting 14:38:08 (40536): No heartbeat from core client for 30 sec - exiting 14:38:09 (40536): No heartbeat from core client for 30 sec - exiting 14:38:10 (40536): No heartbeat from core client for 30 sec - exiting 14:38:11 (40536): No heartbeat from core client for 30 sec - exiting 14:38:12 (40536): No heartbeat from core client for 30 sec - exiting 14:38:13 (40536): No heartbeat from core client for 30 sec - exiting 14:57:46 (40651): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:57:48 (40651): No heartbeat from core client for 30 sec - exiting 15:12:35 (40795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:37 (40795): No heartbeat from core client for 30 sec - exiting 15:12:38 (40795): No heartbeat from core client for 30 sec - exiting 15:12:39 (40795): No heartbeat from core client for 30 sec - exiting 15:26:37 (40927): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:26:40 (40927): No heartbeat from core client for 30 sec - exiting 15:47:58 (41030): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:55:46 (41196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:06:25 (41262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:13:36 (41363): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:14 (41426): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:24 (41533): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:28 (41533): No heartbeat from core client for 30 sec - exiting 16:49:29 (41533): No heartbeat from core client for 30 sec - exiting 16:49:30 (41533): No heartbeat from core client for 30 sec - exiting 17:01:43 (41710): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:45 (41710): No heartbeat from core client for 30 sec - exiting 17:01:46 (41710): No heartbeat from core client for 30 sec - exiting 17:01:47 (41710): No heartbeat from core client for 30 sec - exiting 17:01:48 (41710): No heartbeat from core client for 30 sec - exiting 17:01:49 (41710): No heartbeat from core client for 30 sec - exiting 17:01:50 (41710): No heartbeat from core client for 30 sec - exiting 17:01:51 (41710): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Jun 2011 22:46:54	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	233,280	465,778	1.9966
26 Jun 2011 23:56:19	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	207,360	415,588	2.0042
26 Jun 2011 06:48:53	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	181,440	362,825	1.9997
25 Jun 2011 11:13:57	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	155,520	311,280	2.0015
24 Jun 2011 14:44:22	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	129,600	259,404	2.0016
23 Jun 2011 17:52:57	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	103,680	207,545	2.0018
22 Jun 2011 22:23:16	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	77,760	155,376	1.9981
22 Jun 2011 07:01:34	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	51,840	102,997	1.9868
21 Jun 2011 15:54:24	1006135	12992382	hadcm3n_s72i_1940_40_007300749_1	25,920	51,646	1.9925