Task 15514877

Name	hadcm3n_zlxa_1920_40_008271390_1
Workunit	8426514
Created	28 Dec 2012, 11:40:29 UTC
Sent	28 Dec 2012, 11:44:14 UTC
Report deadline	29 Mar 2013, 19:11:25 UTC
Received	9 Feb 2013, 13:35:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1260615
Run time	9 days 1 hours 39 min 42 sec
CPU time	8 days 20 hours 30 min 52 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4328, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zlxako.pjc2c10 Error converting file to netcdf: dataout/zlxako.pic2c10 Error converting file to netcdf: dataout/zlxako.pfc2c10 Error converting file to netcdf: dataout/zlxaka.phc2c10 Error converting file to netcdf: dataout/zlxaka.pgc2c10 Error converting file to netcdf: dataout/zlxaka.pec2c10 Error converting file to netcdf: dataout/zlxaka.pdc2c10 Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to netcdf error 28 - No space left on device Skipping gmts_generator due to nBUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Feb 2013 13:40:16	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	777,600	765,052	0.9839
08 Feb 2013 19:05:35	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	751,680	749,008	0.9964
06 Feb 2013 23:39:52	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	725,760	731,582	1.0080
05 Feb 2013 18:34:29	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	699,840	713,120	1.0190
04 Feb 2013 17:51:41	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	673,920	694,716	1.0309
03 Feb 2013 19:06:25	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	648,000	676,494	1.0440
03 Feb 2013 13:45:23	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	622,080	658,075	1.0579
03 Feb 2013 04:56:47	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	596,160	638,756	1.0715
02 Feb 2013 22:45:06	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	570,240	618,399	1.0845
02 Feb 2013 17:11:16	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	544,320	597,718	1.0981
25 Jan 2013 18:11:23	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	518,400	577,246	1.1135
22 Jan 2013 19:38:26	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	492,480	556,814	1.1306
21 Jan 2013 17:46:25	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	466,560	536,037	1.1489
20 Jan 2013 15:50:57	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	440,640	515,430	1.1697
20 Jan 2013 02:53:34	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	414,720	494,863	1.1932
19 Jan 2013 17:31:18	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	388,800	474,275	1.2198
19 Jan 2013 11:40:43	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	362,880	453,770	1.2505
16 Jan 2013 22:42:22	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	336,960	433,185	1.2856
15 Jan 2013 17:26:24	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	311,040	412,714	1.3269
13 Jan 2013 15:25:27	1260615	15514877	hadcm3n_zlxa_1920_40_008271390_1	285,120	388,905	1.3640