Task 13664494

Name	hadcm3n_yl8o_1900_40_007514030_4
Workunit	7711505
Created	27 Nov 2011, 10:31:53 UTC
Sent	27 Nov 2011, 10:31:57 UTC
Report deadline	26 Feb 2012, 17:59:08 UTC
Received	28 Dec 2011, 3:38:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1128786
Run time	6 days 8 hours 48 min 10 sec
CPU time	5 days 23 hours 24 min 44 sec
Validate state	Invalid
Credit	4,354.56
Device peak FLOPS	2.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:05:12 (7192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:13 (7192): No heartbeat from core client for 30 sec - exiting 04:05:14 (7192): No heartbeat from core client for 30 sec - exiting 04:05:16 (7192): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:49:26 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:39 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:02:36 (9480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:36:00 (7124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:04:52 (10776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:02 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:27:37 (6336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:29:21 (7756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:01:08 (10900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:28:04 (2736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:06 (2736): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:04:06 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:04:07 (3776): No heartbeat from core client for 30 sec - exiting 04:04:08 (3776): No heartbeat from core client for 30 sec - exiting 04:04:09 (3776): No heartbeat from core client for 30 sec - exiting 04:04:10 (3776): No heartbeat from core client for 30 sec - exiting 04:04:11 (3776): No heartbeat from core client for 30 sec - exiting 04:04:12 (3776): No heartbeat from core client for 30 sec - exiting 04:04:13 (3776): No heartbeat from core client for 30 sec - exiting 04:04:14 (3776): No heartbeat from core client for 30 sec - exiting 04:04:15 (3776): No heartbeat from core client for 30 sec - exiting 04:04:16 (3776): No heartbeat from core client for 30 sec - exiting 04:06:08 (8120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:24 (8120): No heartbeat from core client for 30 sec - exiting 04:06:25 (8120): No heartbeat from core client for 30 sec - exiting 04:06:26 (8120): No heartbeat from core client for 30 sec - exiting 04:06:27 (8120): No heartbeat from core client for 30 sec - exiting 04:06:28 (8120): No heartbeat from core client for 30 sec - exiting 04:06:29 (8120): No heartbeat from core client for 30 sec - exiting 04:06:30 (8120): No heartbeat from core client for 30 sec - exiting 04:06:31 (8120): No heartbeat from core client for 30 sec - exiting 04:06:32 (8120): No heartbeat from core client for 30 sec - exiting 04:06:33 (8120): No heartbeat from core client for 30 sec - exiting 04:07:27 (8680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:44 (9632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:49 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:10:57 (8044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:22 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:19 (7196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:14:22 (6844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:15:22 (7256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:16:44 (7012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:17:23 (8496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:37 (9564): No heartbeat from core client for 30 sec - exiting tmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - No 'heartbeat' from BOINC... 04:20:25 (8204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:37 (8204): No heartbeat from core client for 30 sec - exiting 04:20:38 (8204): No heartbeat from core client for 30 sec - exiting 04:20:39 (8204): No heartbeat from core client for 30 sec - exiting 04:20:40 (8204): No heartbeat from core client for 30 sec - exiting 04:20:41 (8204): No heartbeat from core client for 30 sec - exiting 04:20:42 (8204): No heartbeat from core client for 30 sec - exiting 04:20:43 (8204): No heartbeat from core client for 30 sec - exiting 04:20:44 (8204): No heartbeat from core client for 30 sec - exiting 04:20:45 (8204): No heartbeat from core client for 30 sec - exiting 04:20:46 (8204): No heartbeat from core client for 30 sec - exiting 04:21:38 (7464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:40 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:23:25 (6472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:06 (8400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:25:15 (8232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:25:20 (8232): No heartbeat from core client for 30 sec - exiting 04:25:21 (8232): No heartbeat from core client for 30 sec - exiting 04:25:22 (8232): No heartbeat from core client for 30 sec - exiting 04:25:23 (8232): No heartbeat from core client for 30 sec - exiting 04:25:24 (8232): No heartbeat from core client for 30 sec - exiting 04:27:15 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:50 (7784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:31 (7016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:51 (9004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:58 (9004): No heartbeat from core client for 30 sec - exiting 04:31:59 (9004): No heartbeat from core client for 30 sec - exiting 04:32:00 (9004): No heartbeat from core client for 30 sec - exiting 04:32:01 (9004): No heartbeat from core client for 30 sec - exiting 04:32:02 (9004): No heartbeat from core client for 30 sec - exiting 04:32:03 (9004): No heartbeat from core client for 30 sec - exiting 04:32:04 (9004): No heartbeat from core client for 30 sec - exiting 04:32:05 (9004): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 04:33:15 (6968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 04:36:17 (10196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:18 (10196): No heartbeat from core client for 30 sec - exiting 04:36:19 (10196): No heartbeat from core client for 30 sec - exiting 04:36:20 (10196): No heartbeat from core client for 30 sec - exiting 04:37:49 (9964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:37:58 (9964): No heartbeat from core client for 30 sec - exiting 04:37:59 (9964): No heartbeat from core client for 30 sec - exiting 04:38:00 (9964): No heartbeat from core client for 30 sec - exiting 04:38:01 (9964): No heartbeat from core client for 30 sec - exiting 04:38:02 (9964): No heartbeat from core client for 30 sec - exiting 04:38:03 (9964): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:45:17 (8756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:40:50 (7660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:41:30 (9492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8088, iMonCtr=1 Model crash detected, will try to restart... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8088, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 04:49:24	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	362,880	499,553	1.3766
25 Dec 2011 18:51:06	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	336,960	464,071	1.3772
25 Dec 2011 07:38:42	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	311,040	428,842	1.3787
24 Dec 2011 21:15:47	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	285,120	393,992	1.3818
24 Dec 2011 08:54:57	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	259,200	359,034	1.3852
19 Dec 2011 07:22:20	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	233,280	323,924	1.3886
18 Dec 2011 18:48:51	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	207,360	287,721	1.3875
18 Dec 2011 07:47:20	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	181,440	252,125	1.3896
17 Dec 2011 17:20:56	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	155,520	215,494	1.3856
17 Dec 2011 02:27:18	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	129,600	178,724	1.3790
16 Dec 2011 14:53:26	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	103,680	142,153	1.3711
12 Dec 2011 03:11:56	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	77,760	105,935	1.3623
09 Dec 2011 05:26:49	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	51,840	70,492	1.3598
03 Dec 2011 17:38:44	1128786	13664494	hadcm3n_yl8o_1900_40_007514030_4	25,920	35,713	1.3778