Task 15694066

Name	hadcm3n_o2ik_2140_40_008270312_3
Workunit	8425436
Created	29 Mar 2013, 22:01:16 UTC
Sent	29 Mar 2013, 22:02:43 UTC
Report deadline	29 Jun 2013, 5:29:54 UTC
Received	14 Apr 2013, 2:52:20 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1228594
Run time	15 days 3 hours 18 min 40 sec
CPU time	13 days 3 hours 43 min 53 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:54:26 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... MainError: 07:06:40 AM No files match the supplied pattern. MainError: 07:06:40 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 07:47:17 PM No files match the supplied pattern. MainError: 07:47:17 PM No files match the supplied pattern. 21:57:38 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:04:42 AM No files match the supplied pattern. MainError: 08:04:42 AM No files match the supplied pattern. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1532, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1532, iMonCtr=1 Model crash detected, will try to restart... 03:25:11 (3136): No heartbeat from core client for 30 sec - exiting 03:25:12 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:08:35 PM No files match the supplied pattern. MainError: 08:08:36 PM No files match the supplied pattern. MainError: 08:09:44 AM No files match the supplied pattern. MainError: 08:09:44 AM No files match the supplied pattern. MainError: 08:30:40 PM No files match the supplied pattern. MainError: 08:30:40 PM No files match the supplied pattern. MainError: 06:17:31 AM No files match the supplied pattern. MainError: 06:17:31 AM No files match the supplied pattern. MainError: 01:53:40 PM No files match the supplied pattern. MainError: 01:53:40 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:40:45 PM No files match the supplied pattern. MainError: 11:40:45 PM No files match the supplied pattern. MainError: 01:02:16 PM No files match the supplied pattern. MainError: 01:02:16 PM No files match the supplied pattern. Error converting file to netcdf: dataout/o2ikka.ph11c10 Error converting file to netcdf: dataout/o2ikka.pg11c10 Error converting file to netcdf: dataout/o2ikka.pe11c10 MainError: 12:58:05 AM No files match the supplied pattern. MainError: 12:58:05 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Apr 2013 00:58:46	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	777,600	1,241,353	1.5964
13 Apr 2013 13:04:52	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	751,680	1,199,410	1.5956
12 Apr 2013 23:45:13	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	725,760	1,154,888	1.5913
12 Apr 2013 18:25:03	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	699,840	1,121,357	1.6023
12 Apr 2013 18:25:03	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	673,920	1,094,237	1.6237
12 Apr 2013 18:25:03	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	648,000	1,059,449	1.6350
11 Apr 2013 08:13:51	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	622,080	1,016,126	1.6334
10 Apr 2013 20:13:33	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	596,160	973,425	1.6328
10 Apr 2013 08:28:04	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	570,240	931,246	1.6331
09 Apr 2013 19:51:38	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	544,320	889,002	1.6332
09 Apr 2013 07:09:24	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	518,400	845,853	1.6317
08 Apr 2013 18:41:38	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	492,480	803,218	1.6310
08 Apr 2013 06:06:07	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	466,560	760,436	1.6299
07 Apr 2013 17:36:31	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	440,640	717,783	1.6290
07 Apr 2013 05:01:41	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	414,720	674,553	1.6265
06 Apr 2013 16:46:21	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	388,800	632,334	1.6264
06 Apr 2013 03:21:28	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	362,880	586,906	1.6174
05 Apr 2013 14:30:52	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	336,960	542,775	1.6108
05 Apr 2013 02:07:05	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	311,040	500,573	1.6094
04 Apr 2013 13:45:07	1228594	15694066	hadcm3n_o2ik_2140_40_008270312_3	285,120	458,226	1.6071