Task 13466891

Name	hadcm3n_t411_1940_40_007446235_2
Workunit	7643738
Created	8 Oct 2011, 2:26:10 UTC
Sent	8 Oct 2011, 2:26:37 UTC
Report deadline	7 Jan 2012, 9:53:48 UTC
Received	25 Oct 2011, 17:27:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1090056
Run time	9 days 17 hours 3 min 18 sec
CPU time	9 days 13 hours 3 min 32 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	2.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=784, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t411ko.pje2c10 Error converting file to netcdf: dataout/t411ko.pie2c10 Error converting file to netcdf: dataout/t411ko.pfe2c10 Error converting file to netcdf: dataout/t411ka.phe2c10 Error converting file to netcdf: dataout/t411ka.pge2c10 Error converting file to netcdf: dataout/t411ka.pee2c10 Error converting file to netcdf: dataout/t411ka.pde2c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2584, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 116 - Return code = 16 Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 14:58:57	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	388,800	774,299	1.9915
31 Oct 2011 14:58:57	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	362,880	721,992	1.9896
31 Oct 2011 14:58:57	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	336,960	669,547	1.9870
19 Oct 2011 09:33:02	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	311,040	617,578	1.9855
18 Oct 2011 14:57:18	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	285,120	564,917	1.9813
18 Oct 2011 00:21:39	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	259,200	512,336	1.9766
17 Oct 2011 02:25:07	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	233,280	454,590	1.9487
16 Oct 2011 12:14:39	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	207,360	403,936	1.9480
15 Oct 2011 21:42:08	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	181,440	352,544	1.9430
15 Oct 2011 06:15:04	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	155,520	301,551	1.9390
14 Oct 2011 16:16:50	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	129,600	250,454	1.9325
14 Oct 2011 01:27:44	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	103,680	199,495	1.9241
13 Oct 2011 11:17:54	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	77,760	148,963	1.9157
12 Oct 2011 20:49:54	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	51,840	98,824	1.9063
09 Oct 2011 23:45:39	1090056	13466891	hadcm3n_t411_1940_40_007446235_2	25,920	50,731	1.9572