Task 13025011

Name	hadcm3n_t4ze_1940_40_007315934_0
Workunit	7513364
Created	29 Jun 2011, 0:11:54 UTC
Sent	29 Jun 2011, 0:21:10 UTC
Report deadline	28 Sep 2011, 7:48:21 UTC
Received	13 Aug 2011, 18:45:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125581
Run time	23 days 1 hours 43 min 54 sec
CPU time	20 days 0 hours 18 min 26 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	2.36 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:29:18 (11676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:23 (12240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:38:26 (9352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:36 (10344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:41 (7120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:58 (9228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:11 (10576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:50 (8048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:34:30 (9992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:16:42 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:52 (3212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:55 (1208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:05 (4268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:08 (888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:35 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:50:37 (4232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:11:45 (10216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:05:32 (6564): No heartbeat from core client for 30 sec - exiting 11:05:33 (6564): No heartbeat from core client for 30 sec - exiting 11:05:34 (6564): No heartbeat from core client for 30 sec - exiting 11:05:35 (6564): No heartbeat from core client for 30 sec - exiting 11:05:36 (6564): No heartbeat from core client for 30 sec - exiting 11:05:37 (6564): No heartbeat from core client for 30 sec - exiting 11:05:38 (6564): No heartbeat from core client for 30 sec - exiting 11:05:39 (6564): No heartbeat from core client for 30 sec - exiting 11:05:40 (6564): No heartbeat from core client for 30 sec - exiting 11:05:41 (6564): No heartbeat from core client for 30 sec - exiting 11:05:42 (6564): No heartbeat from core client for 30 sec - exiting 11:05:43 (6564): No heartbeat from core client for 30 sec - exiting 11:05:44 (6564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5568, iMonCtr=1 Model crash detected, will try to restart... 06:36:53 (5436): No heartbeat from core client for 30 sec - exiting 06:36:54 (5436): No heartbeat from core client for 30 sec - exiting 06:36:55 (5436): No heartbeat from core client for 30 sec - exiting 06:36:56 (5436): No heartbeat from core client for 30 sec - exiting 06:36:57 (5436): No heartbeat from core client for 30 sec - exiting 06:36:58 (5436): No heartbeat from core client for 30 sec - exiting 06:36:59 (5436): No heartbeat from core client for 30 sec - exiting 06:37:00 (5436): No heartbeat from core client for 30 sec - exiting 06:37:01 (5436): No heartbeat from core client for 30 sec - exiting 06:37:02 (5436): No heartbeat from core client for 30 sec - exiting 06:37:03 (5436): No heartbeat from core client for 30 sec - exiting 06:37:04 (5436): No heartbeat from core client for 30 sec - exiting 06:37:05 (5436): No heartbeat from core client for 30 sec - exiting 06:37:06 (5436): No heartbeat from core client for 30 sec - exiting 06:37:07 (5436): No heartbeat from core client for 30 sec - exiting 06:37:08 (5436): No heartbeat from core client for 30 sec - exiting 06:37:09 (5436): No heartbeat from core client for 30 sec - exiting 06:37:10 (5436): No heartbeat from core client for 30 sec - exiting 06:37:11 (5436): No heartbeat from core client for 30 sec - exiting 06:37:12 (5436): No heartbeat from core client for 30 sec - exiting 06:37:13 (5436): No heartbeat from core client for 30 sec - exiting 06:37:14 (5436): No heartbeat from core client for 30 sec - exiting 06:37:15 (5436): No heartbeat from core client for 30 sec - exiting 06:37:16 (5436): No heartbeat from core client for 30 sec - exiting 06:37:17 (5436): No heartbeat from core client for 30 sec - exiting 06:37:18 (5436): No heartbeat from core client for 30 sec - exiting 06:37:19 (5436): No heartbeat from core client for 30 sec - exiting 06:37:20 (5436): No heartbeat from core client for 30 sec - exiting 06:37:21 (5436): No heartbeat from core client for 30 sec - exiting 06:37:22 (5436): No heartbeat from core client for 30 sec - exiting 06:37:23 (5436): No heartbeat from core client for 30 sec - exiting 06:37:24 (5436): No heartbeat from core client for 30 sec - exiting 06:37:25 (5436): No heartbeat from core client for 30 sec - exiting 06:37:26 (5436): No heartbeat from core client for 30 sec - exiting 06:37:27 (5436): No heartbeat from core client for 30 sec - exiting 06:37:28 (5436): No heartbeat from core client for 30 sec - exiting 06:37:29 (5436): No heartbeat from core client for 30 sec - exiting 06:37:30 (5436): No heartbeat from core client for 30 sec - exiting 06:37:31 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:37:32 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:10:50 (11328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:06:59 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7444, iMonCtr=1 Model crash detected, will try to restart... 07:49:54 (6244): No heartbeat from core client for 30 sec - exiting 07:49:55 (6244): No heartbeat from core client for 30 sec - exiting 07:49:56 (6244): No heartbeat from core client for 30 sec - exiting 07:49:57 (6244): No heartbeat from core client for 30 sec - exiting 07:49:59 (6244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:00 (6244): No heartbeat from core client for 30 sec - exiting Ocean Restart file copy failed on t4zeko.daf66i0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:06:45 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:19:26 (1000): No heartbeat from core client for 30 sec - exiting 08:19:27 (1000): No heartbeat from core client for 30 sec - exiting 08:19:28 (1000): No heartbeat from core client for 30 sec - exiting 08:19:29 (1000): No heartbeat from core client for 30 sec - exiting 08:19:30 (1000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:56:33 (4000): No heartbeat from core client for 30 sec - exiting 08:56:34 (4000): No heartbeat from core client for 30 sec - exiting 08:56:35 (4000): No heartbeat from core client for 30 sec - exiting 08:56:36 (4000): No heartbeat from core client for 30 sec - exiting 08:56:37 (4000): No heartbeat from core client for 30 sec - exiting 08:56:38 (4000): No heartbeat from core client for 30 sec - exiting 08:56:39 (4000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:12 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:14 (4880): No heartbeat from core client for 30 sec - exiting 09:43:15 (4880): No heartbeat from core client for 30 sec - exiting 09:44:18 (2196): No heartbeat from core client for 30 sec - exiting 09:44:19 (2196): No heartbeat from core client for 30 sec - exiting 09:44:20 (2196): No heartbeat from core client for 30 sec - exiting 09:44:21 (2196): No heartbeat from core client for 30 sec - exiting 09:44:22 (2196): No heartbeat from core client for 30 sec - exiting 09:44:23 (2196): No heartbeat from core client for 30 sec - exiting 09:44:24 (2196): No heartbeat from core client for 30 sec - exiting 09:44:25 (2196): No heartbeat from core client for 30 sec - exiting 09:44:26 (2196): No heartbeat from core client for 30 sec - exiting 09:44:27 (2196): No heartbeat from core client for 30 sec - exiting 09:44:28 (2196): No heartbeat from core client for 30 sec - exiting 09:44:29 (2196): No heartbeat from core client for 30 sec - exiting 09:44:30 (2196): No heartbeat from core client for 30 sec - exiting 09:44:31 (2196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:49:26 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:52:06 (6344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:09:10 (11184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:18:02 (6376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6360, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:54:08 (784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:39:23 (8808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5740, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5740, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5740, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Aug 2011 00:47:20	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	984,960	1,725,688	1.7520
11 Aug 2011 18:29:15	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	959,040	1,681,628	1.7534
10 Aug 2011 21:49:10	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	933,120	1,637,714	1.7551
09 Aug 2011 05:18:06	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	907,200	1,593,543	1.7566
08 Aug 2011 05:58:25	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	881,280	1,549,954	1.7588
06 Aug 2011 20:28:39	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	855,360	1,505,725	1.7603
06 Aug 2011 07:07:00	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	829,440	1,462,790	1.7636
05 Aug 2011 02:54:09	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	803,520	1,418,037	1.7648
04 Aug 2011 03:09:39	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	777,600	1,374,280	1.7673
03 Aug 2011 11:02:51	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	751,680	1,330,623	1.7702
02 Aug 2011 03:36:22	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	725,760	1,286,843	1.7731
01 Aug 2011 07:54:18	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	699,840	1,242,793	1.7758
31 Jul 2011 14:48:50	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	673,920	1,199,007	1.7792
30 Jul 2011 23:42:43	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	648,000	1,155,222	1.7828
28 Jul 2011 01:10:35	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	622,080	1,111,134	1.7862
27 Jul 2011 07:24:54	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	596,160	1,065,831	1.7878
26 Jul 2011 15:30:21	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	570,240	1,019,326	1.7875
25 Jul 2011 23:26:46	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	544,320	972,222	1.7861
25 Jul 2011 22:27:44	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	518,400	925,934	1.7861
25 Jul 2011 21:02:36	1125581	13025011	hadcm3n_t4ze_1940_40_007315934_0	492,480	879,301	1.7855