Task 13021833

Name	hadcm3n_t2b6_1940_40_007314595_0
Workunit	7512025
Created	28 Jun 2011, 16:43:31 UTC
Sent	28 Jun 2011, 16:48:39 UTC
Report deadline	28 Sep 2011, 0:15:50 UTC
Received	9 Jul 2011, 0:50:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153338
Run time	3 days 17 hours 23 min 7 sec
CPU time	3 days 13 hours 22 min 39 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 02:55:28 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t2b6ko.pje3c10 Error converting file to netcdf: dataout/t2b6ko.pie3c10 Error converting file to netcdf: dataout/t2b6ko.pfe3c10 Error converting file to netcdf: dataout/t2b6ka.phe3c10 Error converting file to netcdf: dataout/t2b6ka.pge3c10 Error converting file to netcdf: dataout/t2b6ka.pee3c10 Error converting file to netcdf: dataout/t2b6ka.pde3c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t2b6ko.pje3c10 Error converting file to netcdf: dataout/t2b6ko.pie3c10 Error converting file to netcdf: dataout/t2b6ko.pfe3c10 Error converting file to netcdf: dataout/t2b6ka.phe3c10 Error converting file to netcdf: dataout/t2b6ka.pge3c10 Error converting file to netcdf: dataout/t2b6ka.pee3c10 Error converting file to netcdf: dataout/t2b6ka.pde3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on t2b6ko.dae96l0 Ocean Restart file copy failed on t2b6ko.dae96m0 Ocean Restart file copy failed on t2b6ko.dae96n0 Ocean Restart file copy failed on t2b6ko.dae96o0 Ocean Restart file copy failed on t2b6ko.dae96p0 Ocean Restart file copy failed on t2b6ko.dae96q0 Ocean Restart file copy failed on t2b6ko.dae96r0 Ocean Restart file copy failed on t2b6ko.dae96s0 Ocean Restart file copy failed on t2b6ko.dae96t0 Ocean Restart file copy failed on t2b6ko.dae96u0 Ocean Restart file copy failed on t2b6ko.dae9710 Ocean Restart file copy failed on t2b6ko.dae9720 Ocean Restart file copy failed on t2b6ko.dae9730 Ocean Restart file copy failed on t2b6ko.dae9740 Ocean Restart file copy failed on t2b6ko.dae9750 Ocean Restart file copy failed on t2b6ko.dae9760 Ocean Restart file copy failed on t2b6ko.dae9770 Ocean Restart file copy failed on t2b6ko.dae9780 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2011 20:18:56	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	207,360	287,579	1.3869
07 Jul 2011 15:38:32	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	181,440	252,065	1.3892
05 Jul 2011 17:27:52	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	155,520	215,883	1.3881
04 Jul 2011 00:52:48	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	129,600	179,449	1.3846
02 Jul 2011 18:10:30	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	103,680	143,847	1.3874
01 Jul 2011 22:03:52	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	77,760	107,661	1.3845
01 Jul 2011 17:29:40	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	51,840	71,549	1.3802
29 Jun 2011 15:47:40	1153338	13021833	hadcm3n_t2b6_1940_40_007314595_0	25,920	35,871	1.3839