Task 19327236

Name	hadcm3n_ldl1_198012_480_350_010332274_1
Workunit	10332274
Created	9 Mar 2016, 5:47:18 UTC
Sent	9 Mar 2016, 5:47:48 UTC
Report deadline	19 Feb 2017, 11:07:48 UTC
Received	13 Apr 2016, 2:01:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1351468
Run time	21 days 12 hours 3 min 40 sec
CPU time	11 days 8 hours 41 min 18 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish 04:20:14 (1243): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:37:50 (1387): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:17:01 (2328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:25:40 (17699): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:25:42 (17699): No heartbeat from core client for 30 sec - exiting 13:31:32 (69915): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:56 (17879): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:15:57 (47795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:13:14 (1448): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:28:55 (1509): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:58:38 (2358): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:21:43 (87294): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:58 (20439): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:23:01 (20439): No heartbeat from core client for 30 sec - exiting 04:52:11 (42063): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:16:52 (1413): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:08:48 (87939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:43:26 (47959): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:49:08 (59594): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:39 (60277): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:56:15 (60766): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 05:31:23 (87702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:25:26 (89255): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:59:00 (1415): No heartbeat from core client for 30 sec - exiting 10:59:01 (1415): No heartbeat from core client for 30 sec - exiting 10:59:02 (1415): No heartbeat from core client for 30 sec - exiting 10:59:03 (1415): No heartbeat from core client for 30 sec - exiting 10:59:04 (1415): No heartbeat from core client for 30 sec - exiting 10:59:05 (1415): No heartbeat from core client for 30 sec - exiting 10:59:06 (1415): No heartbeat from core client for 30 sec - exiting 10:59:07 (1415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /var/db/boinc/projects/climateprediction.net/hadcm3n_ldl1_198012_480_350_010332274/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Apr 2016 12:08:06	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	1,036,800	981,619	0.9468
11 Apr 2016 02:32:40	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	1,010,880	938,530	0.9284
10 Apr 2016 12:39:35	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	984,960	895,234	0.9089
09 Apr 2016 20:31:37	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	959,040	850,933	0.8873
09 Apr 2016 01:54:50	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	933,120	806,240	0.8640
08 Apr 2016 20:35:35	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	907,200	763,080	0.8411
07 Apr 2016 00:56:45	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	881,280	719,673	0.8166
06 Apr 2016 04:31:04	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	855,360	676,467	0.7909
05 Apr 2016 13:57:11	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	829,440	695,988	0.8391
05 Apr 2016 02:22:08	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	803,520	653,059	0.8127
02 Apr 2016 06:05:58	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	777,600	742,186	0.9545
31 Mar 2016 10:29:43	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	751,680	699,225	0.9302
30 Mar 2016 22:13:38	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	725,760	655,307	0.9029
30 Mar 2016 04:21:09	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	699,840	637,613	0.9111
29 Mar 2016 05:01:11	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	673,920	594,915	0.8828
28 Mar 2016 02:40:33	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	648,000	551,846	0.8516
27 Mar 2016 09:59:27	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	622,080	508,770	0.8179
26 Mar 2016 09:51:27	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	596,160	464,906	0.7798
25 Mar 2016 11:16:43	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	570,240	439,812	0.7713
24 Mar 2016 08:27:12	1351468	19327236	hadcm3n_ldl1_198012_480_350_010332274_1	544,320	396,484	0.7284