Task 13263966

Name	hadcm3n_t1cp_1980_40_007411623_1
Workunit	7609253
Created	16 Aug 2011, 21:13:30 UTC
Sent	16 Aug 2011, 21:18:45 UTC
Report deadline	16 Nov 2011, 4:45:56 UTC
Received	19 Sep 2011, 8:53:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1154437
Run time	18 days 18 hours 29 min 5 sec
CPU time	17 days 5 hours 14 min 56 sec
Validate state	Invalid
Credit	11,508.48
Device peak FLOPS	2.78 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x1057c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x1057c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x103c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x103c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x1057c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(67394,0xa05fb540) malloc: * error for object 0x1057c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=67396, selfPID=67396, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x181f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x85ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68143,0xa05fb540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x1044604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(68337,0xa05fb540) malloc: * error for object 0x102de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 3 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Sep 2011 11:39:48	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	959,040	1,472,102	1.5350
17 Sep 2011 12:17:03	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	933,120	1,431,853	1.5345
16 Sep 2011 16:17:02	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	907,200	1,391,273	1.5336
13 Sep 2011 08:59:55	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	881,280	1,351,471	1.5335
12 Sep 2011 20:08:58	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	855,360	1,311,501	1.5333
12 Sep 2011 09:52:21	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	829,440	1,271,301	1.5327
12 Sep 2011 09:52:21	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	803,520	1,229,808	1.5305
12 Sep 2011 09:52:21	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	777,600	1,188,272	1.5281
12 Sep 2011 09:52:21	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	751,680	1,146,735	1.5256
09 Sep 2011 23:11:03	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	725,760	1,105,159	1.5228
09 Sep 2011 23:11:03	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	699,840	1,063,640	1.5198
09 Sep 2011 23:11:02	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	673,920	1,022,186	1.5168
08 Sep 2011 10:32:05	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	648,000	980,566	1.5132
08 Sep 2011 10:32:05	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	622,080	940,049	1.5111
05 Sep 2011 09:11:16	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	596,160	898,934	1.5079
05 Sep 2011 09:11:15	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	570,240	860,769	1.5095
05 Sep 2011 09:11:15	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	544,320	823,666	1.5132
02 Sep 2011 21:43:04	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	518,400	784,902	1.5141
02 Sep 2011 14:08:46	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	492,480	744,412	1.5116
31 Aug 2011 09:36:28	1154437	13263966	hadcm3n_t1cp_1980_40_007411623_1	466,560	705,899	1.5130