Task 12348389

Name	famous_vdtn_799_200_006703681_5
Workunit	6906934
Created	28 Nov 2010, 16:04:20 UTC
Sent	28 Nov 2010, 16:06:12 UTC
Report deadline	27 Feb 2011, 23:33:23 UTC
Received	7 Dec 2010, 10:53:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1049277
Run time	8 days 14 hours 24 min 12 sec
CPU time	39 min 33 sec
Validate state	Invalid
Credit	2,007.39
Device peak FLOPS	2.59 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: STASH : Unsupported MPP submodel tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_se_6.11_windows_intelx86.dll cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_um_6.11_windows_intelx86.exe BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: STASH : Unsupported MPP submodel tmp/pipe_dummy : Not enough space BUFFIN: C I/O Error ferror - Unit Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vdtn_799_200_006703681/jobs/afyel.namelists BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: STASH : Unsupported MPP submodel tmp/pipe_dummy forrtl: Not enough storage is available to process this command. forrtl: severe (39): error during read, unit 22, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vdtn_799_200_006703681\datain\ancil\ctldata\STASHmaster\STASHmaster_A Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15752, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy forrtl: Not enough storage is available to process this command. forrtl: severe (39): error during read, unit 22, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vdtn_799_200_006703681\datain\ancil\ctldata\STASHmaster\STASHmaster_A Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15752, iMonCtr=1 Model crash detected, will try to restart... no start tag in app init data 23:50:21 (14104): Can't parse init data file - running in standalone mode no start tag in app init data 23:55:24 (15752): Can't parse init data file - running in standalone mode Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy cpdnmonitor: error reading file dataout/atmos_restart.hold BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vdtn_799_200_006703681/jobs/afyel.namelists cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vdtn_799_200_006703681/dataout/atmos_restart.day cpdnmonitor: error reading file C:\ProgramData\BOINC/projects/climateprediction.net/famous_vdtn_799_200_006703681/dataout/ocean_restart.day no start tag in app init data 00:13:51 (16076): Can't parse init data file - running in standalone mode Model crashed: READHEAD: I/O error tmp/pipe_dummy no start tag in app init data 00:13:54 (15752): Can't parse init data file - running in standalone mode forrtl: severe (39): error during read, unit 22, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vdtn_799_200_006703681\datain\ancil\ctldata\STASHmaster\STASHmaster_O Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15752, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy BUFFIN: Read Failed Model crashed: DRLANDF1 : Error in READ_LAND_SEA. tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy Model crashed: REPLANCO :I\|O error tmp/pipe_dummy Model crashed: READHIST: Read ERROR on history file for namelist NLIHISTO tmp/pipe_dummy forrtl: Not enough storage is available to process this command. forrtl: severe (39): error during read, unit 3, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vdtn_799_200_006703681\datain\ancil\ctldata\stasets\X01003254 Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15752, iMonCtr=1 Model crash detected, will try to restart... forrtl: severe (39): error during read, unit 22, file C:\ProgramData\BOINC\projects\climateprediction.net\famous_vdtn_799_200_006703681\datain\ancil\ctldata\STASHmaster\STASHmaster_O Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15752, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: Read Failed: Not enough space BUFFIN: C I/O Error ferror - Unit 41 - Return code = 32 Model crashed: READ_FLH: I/O error tmp/pipe_dummy no start tag in app init data 00:50:17 (15752): Can't parse init data file - running in standalone mode 00:50:18 (17968): Can't parse init data file - running in standalone mode BUFFOUT: Write Failed: Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy no start tag in app init data 01:00:28 (15752): Can't parse init data file - running in standalone mode Model crashed: REPLANCO :I\|O error tmp/pipe_dummy Model crashed: REPLANCA :I/O ERROR tmp/pipe_dummy : Not enough space BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy Model crashed: INITTIME: Atmosphere data time mismatch tmp/pipe_dummy no start tag in app init data 01:32:13 (3644): Can't parse init data file - running in standalone mode Model crashed: INITTIME: Atmosphere data time mismatch tmp/pipe_dummy no start tag in app init data 01:32:18 (15752): Can't parse init data file - running in standalone mode Model crashed: INITTIME: Atmosphere data time mismatch tmp/pipe_dummy Model crashed: INITTIME: Atmosphere data time mismatch tmp/pipe_dummy Model crashed: INITTIME: Atmosphere data time mismatch tmp/pipe_dummy Sorry, too many model crashes! :-( 01:32:29 (15752): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Dec 2010 10:58:30	1049277	12348389	famous_vdtn_799_200_006703681_5	608,426	2,856	0.0047
07 Dec 2010 10:58:30	1049277	12348389	famous_vdtn_799_200_006703681_5	599,066	232,413	0.3880
07 Dec 2010 10:58:30	1049277	12348389	famous_vdtn_799_200_006703681_5	589,706	227,918	0.3865
07 Dec 2010 07:25:08	1049277	12348389	famous_vdtn_799_200_006703681_5	580,346	224,312	0.3865
06 Dec 2010 23:37:07	1049277	12348389	famous_vdtn_799_200_006703681_5	570,986	220,672	0.3865
06 Dec 2010 21:43:38	1049277	12348389	famous_vdtn_799_200_006703681_5	561,626	217,055	0.3865
06 Dec 2010 19:45:16	1049277	12348389	famous_vdtn_799_200_006703681_5	552,266	213,436	0.3865
06 Dec 2010 17:40:15	1049277	12348389	famous_vdtn_799_200_006703681_5	542,906	209,668	0.3862
06 Dec 2010 15:42:07	1049277	12348389	famous_vdtn_799_200_006703681_5	533,546	206,049	0.3862
06 Dec 2010 13:40:59	1049277	12348389	famous_vdtn_799_200_006703681_5	524,186	202,429	0.3862
06 Dec 2010 11:38:27	1049277	12348389	famous_vdtn_799_200_006703681_5	514,826	198,814	0.3862
06 Dec 2010 09:40:01	1049277	12348389	famous_vdtn_799_200_006703681_5	505,466	195,207	0.3862
06 Dec 2010 08:35:43	1049277	12348389	famous_vdtn_799_200_006703681_5	496,106	191,600	0.3862
06 Dec 2010 08:35:43	1049277	12348389	famous_vdtn_799_200_006703681_5	486,746	188,004	0.3862
06 Dec 2010 08:35:43	1049277	12348389	famous_vdtn_799_200_006703681_5	477,386	184,412	0.3863
06 Dec 2010 08:35:43	1049277	12348389	famous_vdtn_799_200_006703681_5	468,026	180,821	0.3863
05 Dec 2010 23:53:53	1049277	12348389	famous_vdtn_799_200_006703681_5	458,666	177,166	0.3863
05 Dec 2010 22:09:07	1049277	12348389	famous_vdtn_799_200_006703681_5	449,306	173,497	0.3861
05 Dec 2010 20:25:34	1049277	12348389	famous_vdtn_799_200_006703681_5	439,946	169,864	0.3861
05 Dec 2010 18:30:49	1049277	12348389	famous_vdtn_799_200_006703681_5	430,586	166,239	0.3861