Task 15501722

Name	hadcm3n_o0sq_2140_40_008268958_0
Workunit	8424082
Created	23 Dec 2012, 21:31:14 UTC
Sent	23 Dec 2012, 21:42:02 UTC
Report deadline	25 Mar 2013, 5:09:13 UTC
Received	13 Feb 2013, 23:22:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1166383
Run time	50 days 11 hours 42 min 42 sec
CPU time	34 days 18 hours 29 min 10 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4432, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5056, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5440, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4960, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... C09:17:01 (5860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:18:20 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:54:53 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:47 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 02:10:58 AM No files match the supplied pattern. MainError: 02:10:58 AM No files match the supplied pattern. MainError: 12:11:27 AM No files match the supplied pattern. MainError: 12:11:27 AM No files match the supplied pattern. MainError: 09:34:21 PM No files match the supplied pattern. MainError: 09:34:21 PM No files match the supplied pattern. MainError: 07:21:21 PM No files match the supplied pattern. MainError: 07:21:21 PM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3412, iMonCtr=1 Model crash detected, will try to restart... MainError: 04:56:34 PM No files match the supplied pattern. MainError: 04:56:34 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 04:13:37 PM No files match the supplied pattern. MainError: 04:13:37 PM No files match the supplied pattern. MainError: 07:58:44 PM No files match the supplied pattern. MainError: 07:58:44 PM No files match the supplied pattern. MainError: 12:57:58 AM No files match the supplied pattern. MainError: 12:57:58 AM No files match the supplied pattern. MainError: 04:59:38 AM No files match the supplied pattern. MainError: 04:59:38 AM No files match the supplied pattern. MainError: 11:52:18 AM No files match the supplied pattern. MainError: 11:52:18 AM No files match the supplied pattern. Error converting file to netcdf: dataout/o0sqka.ph11c10 Error converting file to netcdf: dataout/o0sqka.pg11c10 Error converting file to netcdf: dataout/o0sqka.pe11c10 MainError: 06:52:02 PM No files match the supplied pattern. MainError: 06:52:02 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Feb 2013 18:55:15	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	777,600	4,129,805	5.3110
11 Feb 2013 11:56:52	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	751,680	3,946,013	5.2496
09 Feb 2013 05:00:56	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	725,760	3,762,104	5.1837
07 Feb 2013 01:00:22	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	699,840	3,586,156	5.1243
04 Feb 2013 20:02:06	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	673,920	3,411,116	5.0616
02 Feb 2013 16:15:58	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	648,000	3,239,088	4.9986
31 Jan 2013 17:01:03	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	622,080	3,078,634	4.9489
30 Jan 2013 19:31:28	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	596,160	3,001,491	5.0347
29 Jan 2013 21:36:39	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	570,240	2,923,262	5.1264
29 Jan 2013 00:13:05	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	544,320	2,846,436	5.2293
28 Jan 2013 02:13:27	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	518,400	2,767,434	5.3384
26 Jan 2013 23:54:05	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	492,480	2,673,109	5.4279
25 Jan 2013 03:09:21	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	466,560	2,518,521	5.3981
23 Jan 2013 03:07:11	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	440,640	2,355,555	5.3458
21 Jan 2013 04:40:23	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	414,720	2,197,202	5.2980
19 Jan 2013 06:19:39	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	388,800	2,041,056	5.2496
17 Jan 2013 08:49:36	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	362,880	1,887,576	5.2017
15 Jan 2013 12:20:05	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	336,960	1,736,236	5.1526
13 Jan 2013 16:00:33	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	311,040	1,586,295	5.1000
11 Jan 2013 22:57:12	1166383	15501722	hadcm3n_o0sq_2140_40_008268958_0	285,120	1,444,660	5.0668