climateprediction.net home page
Task 15863917

Task 15863917

Name hadcm3n_n2yl_1880_40_008375131_3
Workunit 8525990
Created 25 Jun 2013, 16:39:45 UTC
Sent 25 Jun 2013, 17:01:01 UTC
Report deadline 25 Sep 2013, 0:28:12 UTC
Received 23 Sep 2013, 15:41:35 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1169903
Run time 6 days 4 hours 21 min 3 sec
CPU time 5 days 8 hours 11 min 35 sec
Validate state Invalid
Credit 3,110.40
Device peak FLOPS 3.15 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
01:07:02 (21624): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
01:07:38 (6559): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
00:36:27 (81075): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:39:19 (82299): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
hadcm3n_6.07_i686-apple-darwin(5310,0xa0829540) malloc: *** error for object 0x2806c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(5310,0xa0829540) malloc: *** error for object 0x2806c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29778,0xa0829540) malloc: *** error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation
00:02:17 (29778): No heartbeat from core client for 30 sec - exiting
02:21:55 (29778): No heartbeat from core client for 30 sec - exiting
23:02:00 (29778): No heartbeat from core client for 30 sec - exiting
18:25:56 (29778): No heartbeat from core client for 30 sec - exiting
09:33:20 (29778): No heartbeat from core client for 30 sec - exiting
00:18:21 (29778): No heartbeat from core client for 30 sec - exiting
20:20:57 (29778): No heartbeat from core client for 30 sec - exiting
20:37:53 (29778): No heartbeat from core client for 30 sec - exiting
02:01:45 (29778): No heartbeat from core client for 30 sec - exiting
18:46:38 (29778): No heartbeat from core client for 30 sec - exiting
01:37:45 (29778): No heartbeat from core client for 30 sec - exiting
03:40:35 (29778): No heartbeat from core client for 30 sec - exiting
16:26:13 (29778): No heartbeat from core client for 30 sec - exiting
23:09:11 (29778): No heartbeat from core client for 30 sec - exiting
03:04:36 (29778): No heartbeat from core client for 30 sec - exiting
00:06:00 (29778): No heartbeat from core client for 30 sec - exiting
03:43:44 (29778): No heartbeat from core client for 30 sec - exiting
06:50:39 (29778): No heartbeat from core client for 30 sec - exiting
04:52:01 (29778): No heartbeat from core client for 30 sec - exiting
04:52:02 (29778): No heartbeat from core client for 30 sec - exiting
04:52:03 (29778): No heartbeat from core client for 30 sec - exiting
06:26:56 (29778): No heartbeat from core client for 30 sec - exiting
11:58:32 (29778): No heartbeat from core client for 30 sec - exiting
04:04:04 (29778): No heartbeat from core client for 30 sec - exiting
17:56:17 (29778): No heartbeat from core client for 30 sec - exiting
16:38:22 (29778): No heartbeat from core client for 30 sec - exiting
22:29:26 (29778): No heartbeat from core client for 30 sec - exiting
21:58:16 (29778): No heartbeat from core client for 30 sec - exiting
15:28:24 (29778): No heartbeat from core client for 30 sec - exiting
02:03:58 (29778): No heartbeat from core client for 30 sec - exiting
04:24:49 (29778): No heartbeat from core client for 30 sec - exiting
11:01:15 (29778): No heartbeat from core client for 30 sec - exiting
04:56:52 (29778): No heartbeat from core client for 30 sec - exiting
07:50:00 (29778): No heartbeat from core client for 30 sec - exiting
21:35:41 (29778): No heartbeat from core client for 30 sec - exiting
00:55:34 (29778): No heartbeat from core client for 30 sec - exiting
00:55:35 (29778): No heartbeat from core client for 30 sec - exiting
00:14:33 (29778): No heartbeat from core client for 30 sec - exiting
00:14:34 (29778): No heartbeat from core client for 30 sec - exiting
06:31:14 (29778): No heartbeat from core client for 30 sec - exiting
12:43:16 (29778): No heartbeat from core client for 30 sec - exiting
18:32:53 (29778): No heartbeat from core client for 30 sec - exiting
03:25:13 (29778): No heartbeat from core client for 30 sec - exiting
04:26:10 (29778): No heartbeat from core client for 30 sec - exiting
08:33:15 (29778): No heartbeat from core client for 30 sec - exiting
08:36:14 (29778): No heartbeat from core client for 30 sec - exiting
22:07:11 (29778): No heartbeat from core client for 30 sec - exiting
05:37:31 (29778): No heartbeat from core client for 30 sec - exiting
09:37:19 (29778): No heartbeat from core client for 30 sec - exiting
21:18:20 (29778): No heartbeat from core client for 30 sec - exiting
hadcm3n_6.07_i686-apple-darwin(1108,0xa0cb2540) malloc: *** error for object 0x4000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(1108,0xa0cb2540) malloc: *** error for object 0x4000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(1108,0xa0cb2540) malloc: *** error for object 0x4000e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(1108,0xa0cb2540) malloc: *** error for object 0x4000e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(1108,0xa0cb2540) malloc: *** error for object 0x180a600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
SIGSEGV: segmentation violation

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
18:24:40 (1108): No heartbeat from core client for 30 sec - exiting
01:53:00 (1108): No heartbeat from core client for 30 sec - exiting
13:18:18 (1108): No heartbeat from core client for 30 sec - exiting
06:06:31 (1108): No heartbeat from core client for 30 sec - exiting
16:45:55 (1108): No heartbeat from core client for 30 sec - exiting
01:19:55 (1108): No heartbeat from core client for 30 sec - exiting
01:59:31 (1108): No heartbeat from core client for 30 sec - exiting
23:23:36 (1108): No heartbeat from core client for 30 sec - exiting
11:16:15 (1108): No heartbeat from core client for 30 sec - exiting
12:55:58 (1108): No heartbeat from core client for 30 sec - exiting
00:12:36 (1108): No heartbeat from core client for 30 sec - exiting
05:00:42 (1108): No heartbeat from core client for 30 sec - exiting
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x3805c04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x3805c00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x5022600: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(679,0xa05ca540) malloc: *** error for object 0x801c404: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
10:37:39 (679): No heartbeat from core client for 30 sec - exiting
10:37:40 (679): No heartbeat from core client for 30 sec - exiting
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts
CPDN Monitor - No 'heartbeat' from BOINC...
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_n2yl_1880_40_008375131/dataout/ocean_restart.day after 11 attempts

Model crashed: READ_FLH: I/O error                                                                                                                                                                                                                                             tmp/pipe_dummy                                                                  2048    
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
06 Jul 2013 04:37:14 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 259,200 439,305 1.6948
04 Jul 2013 14:26:52 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 233,280 396,244 1.6986
03 Jul 2013 18:12:17 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 207,360 354,333 1.7088
02 Jul 2013 12:08:27 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 181,440 311,783 1.7184
02 Jul 2013 11:53:59 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 155,520 267,959 1.7230
02 Jul 2013 10:48:37 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 129,600 224,283 1.7306
02 Jul 2013 10:24:35 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 103,680 179,957 1.7357
02 Jul 2013 09:57:04 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 77,760 135,851 1.7471
28 Jun 2013 05:37:08 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 51,840 91,492 1.7649
26 Jun 2013 16:23:40 1169903 15863917 hadcm3n_n2yl_1880_40_008375131_3 25,920 45,598 1.7592


©2024 climateprediction.net