Task 15789886

Name	hadcm3n_zjir_1920_40_008369503_1
Workunit	8520362
Created	19 May 2013, 6:50:27 UTC
Sent	19 May 2013, 6:50:45 UTC
Report deadline	18 Aug 2013, 14:17:56 UTC
Received	12 Oct 2013, 2:44:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1281494
Run time	15 days 19 hours 56 min 40 sec
CPU time	15 days 6 hours 15 min 51 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4288, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:10:35 (3436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:36 (3436): No heartbeat from core client for 30 sec - exiting 21:10:37 (3436): No heartbeat from core client for 30 sec - exiting 21:10:38 (3436): No heartbeat from core client for 30 sec - exiting 21:10:39 (3436): No heartbeat from core client for 30 sec - exiting 21:10:40 (3436): No heartbeat from core client for 30 sec - exiting 21:10:41 (3436): No heartbeat from core client for 30 sec - exiting 21:10:42 (3436): No heartbeat from core client for 30 sec - exiting 21:10:43 (3436): No heartbeat from core client for 30 sec - exiting 21:10:44 (3436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6108, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... 11:35:37 (4256): No heartbeat from core client for 30 sec - exiting 11:35:39 (4256): No heartbeat from core client for 30 sec - exiting 11:35:40 (4256): No heartbeat from core client for 30 sec - exiting 11:35:41 (4256): No heartbeat from core client for 30 sec - exiting 11:35:42 (4256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6340, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4376, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:45:21 (3548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1848, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2556, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2220, iMonCtr=1 Model crash detected, will try to restart... 17:23:37 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7224, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5140, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5744, iMonCtr=1 Model crash detected, will try to restart... 17:24:16 (5796): No heartbeat from core client for 30 sec - exiting 17:24:17 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6380, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4572, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zjirko.pjd7c10 Error converting file to netcdf: dataout/zjirko.pid7c10 Error converting file to netcdf: dataout/zjirko.pfd7c10 Error converting file to netcdf: dataout/zjirka.phd7c10 Error converting file to netcdf: dataout/zjirka.pgd7c10 Error converting file to netcdf: dataout/zjirka.ped7c10 Error converting file to netcdf: dataout/zjirka.pdd7c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1260, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5328, iMonCtr=1 Model crash detected, will try to restart... 13:49:42 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6004, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6132, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5824, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5960, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5584, iMonCtr=1 Model crash detected, will try to restart... 15:20:30 (5292): No heartbeat from core client for 30 sec - exiting 15:20:31 (5292): No heartbeat from core client for 30 sec - exiting 15:20:32 (5292): No heartbeat from core client for 30 sec - exiting 15:20:33 (5292): No heartbeat from core client for 30 sec - exiting 15:20:34 (5292): No heartbeat from core client for 30 sec - exiting 15:20:35 (5292): No heartbeat from core client for 30 sec - exiting 15:20:37 (5292): No heartbeat from core client for 30 sec - exiting 15:20:38 (5292): No heartbeat from core client for 30 sec - exiting 15:20:39 (5292): No heartbeat from core client for 30 sec - exiting 15:20:40 (5292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2932, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3760, iMonCtr=1 Model crash detected, will try to restart... 17:44:49 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:50 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Oct 2013 02:47:28	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	777,600	1,318,548	1.6957
03 Oct 2013 00:42:27	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	751,680	1,272,467	1.6928
27 Sep 2013 03:30:38	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	725,760	1,222,955	1.6851
21 Sep 2013 19:45:57	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	699,840	1,173,447	1.6767
15 Sep 2013 19:47:16	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	673,920	1,124,331	1.6683
14 Sep 2013 03:50:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	648,000	1,075,056	1.6590
10 Sep 2013 00:30:48	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	622,080	1,026,129	1.6495
02 Sep 2013 03:57:14	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	596,160	977,745	1.6401
01 Sep 2013 04:02:39	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	570,240	936,399	1.6421
30 Aug 2013 22:29:43	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	544,320	898,830	1.6513
26 Aug 2013 23:07:48	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	518,400	862,384	1.6635
25 Aug 2013 20:05:26	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	492,480	825,418	1.6760
24 Aug 2013 04:01:56	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	466,560	788,311	1.6896
21 Aug 2013 22:25:20	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	440,640	750,559	1.7033
14 Aug 2013 21:30:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	414,720	707,129	1.7051
14 Aug 2013 21:30:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	388,800	662,742	1.7046
14 Aug 2013 21:30:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	362,880	618,592	1.7047
14 Aug 2013 21:30:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	336,960	574,574	1.7052
14 Aug 2013 21:30:53	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	311,040	528,771	1.7000
25 Jul 2013 00:05:48	1281494	15789886	hadcm3n_zjir_1920_40_008369503_1	285,120	479,270	1.6809