Task 15798473

Name	hadcm3n_z9mz_1960_40_008271760_3
Workunit	8426884
Created	27 May 2013, 6:24:52 UTC
Sent	27 May 2013, 6:25:03 UTC
Report deadline	26 Aug 2013, 13:52:14 UTC
Received	12 Sep 2013, 3:38:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1282791
Run time	10 days 4 hours 30 min 29 sec
CPU time	9 days 3 hours 47 min 42 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	2.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1584, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1696, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3588, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3064, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3580, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/z9mzko.pjg9c10 Error converting file to netcdf: dataout/z9mzko.pig9c10 Error converting file to netcdf: dataout/z9mzko.pfg9c10 Error converting file to netcdf: dataout/z9mzka.phg9c10 Error converting file to netcdf: dataout/z9mzka.pgg9c10 Error converting file to netcdf: dataout/z9mzka.peg9c10 Error converting file to netcdf: dataout/z9mzka.pdg9c10 CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4068, iMonCtr=1 Model crash detected, will try to restart... 10:50:09 (2328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4560, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1976, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Aug 2013 07:17:07	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	336,960	740,137	2.1965
27 Aug 2013 01:34:13	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	311,040	683,352	2.1970
16 Aug 2013 06:54:09	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	285,120	626,309	2.1967
15 Aug 2013 01:33:05	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	259,200	568,359	2.1927
15 Aug 2013 01:33:05	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	233,280	509,100	2.1824
15 Aug 2013 01:33:05	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	207,360	448,731	2.1640
24 Jul 2013 02:31:00	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	181,440	390,262	2.1509
23 Jul 2013 19:05:12	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	155,520	333,618	2.1452
27 Jun 2013 05:52:45	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	129,600	276,540	2.1338
21 Jun 2013 01:55:08	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	103,680	218,677	2.1092
19 Jun 2013 00:07:14	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	77,760	161,250	2.0737
07 Jun 2013 05:13:55	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	51,840	104,068	2.0075
30 May 2013 23:02:26	1282791	15798473	hadcm3n_z9mz_1960_40_008271760_3	25,920	50,041	1.9306