Task 13566850

Name	hadcm3n_yeck_1900_40_007525591_4
Workunit	7723066
Created	30 Oct 2011, 4:37:01 UTC
Sent	30 Oct 2011, 5:50:46 UTC
Report deadline	29 Jan 2012, 13:17:57 UTC
Received	5 Nov 2011, 14:57:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1111270
Run time	5 days 11 hours 14 min 30 sec
CPU time	5 days 11 hours 14 min 30 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.2.14</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:08 (20928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:25:08 (20938): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:40:57 (20946): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:05:25 (20966): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:42 (20982): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:31 (21010): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:19 (21021): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:17 (21040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:05 (21078): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:04 (21222): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:54:17 (21431): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:54:27 (21858): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:32 (21899): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:26:06 (21907): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:03:41 (21936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:21 (21977): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:21 (22037): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:11 (22080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:08:46 (22091): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:09:47 (22104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:29 (22305): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:16 (22315): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:34 (22345): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:52:59 (22358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:47 (22368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:22 (22380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:07:23 (22393): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:21 (22403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:42:44 (22424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:47 (22456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:51:30 (22485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:37 (22526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:51 (22538): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:14 (22551): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:46 (22570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:51:02 (23076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:15 (23192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:17 (23208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:45 (23217): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:30 (23227): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:59 (23244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:31:46 (23269): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:17:03 (23329): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:09 (23384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:22 (23392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:27 (23403): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:48:32 (23415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:53:29 (23436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:30 (23445): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:46 (23458): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:02:58 (23471): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:51:31 (23547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:06:27 (23593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:01 (23610): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:54:10 (23657): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:46 (23665): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:58:54 (23725): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:22 (23734): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:12 (23744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:14 (23766): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:07 (23780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:50:18 (23790): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:14 (23810): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:29:59 (23834): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:30 (23853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:50:55 (23872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:47 (23883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:17:55 (23896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:53 (23944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:36 (24076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:47:54 (24098): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:54:58 (24111): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:08:03 (24127): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:28:13 (24151): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:36:37 (24178): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:05 (24192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:15 (24220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:55 (24238): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:16 (24260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:35 (24272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:46:07 (24285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:36 (24312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:24 (24330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:15 (24360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:42:46 (24370): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:57:49 (24397): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:06:38 (24416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:23:23 (24428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:41:44 (24455): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:56:43 (24485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:09:22 (24504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:27 (24526): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:25 (24547): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:37:37 (24565): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:39:09 (24763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:33:21 (24820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0902d640 *** ======= Backtrace: ========= /lib32/libc.so.6[0xf7d8f10c] /lib32/libc.so.6(cfree+0x85)[0xf7d91145] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7f5e6b1] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7f5e70d] hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe5)[0xf7d3842d] hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 fd:00 16641 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 fd:00 16641 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 080e4000 00:00 0 08fd8000-0903d000 rw-p 08fd8000 00:00 0 [heap] f7700000-f7721000 rw-p f7700000 00:00 0 f7721000-f7800000 ---p f7721000 00:00 0 f78a6000-f7d21000 rw-s 00000000 fd:00 120 /var/lib/boinc-client/slots/1/138785 f7d21000-f7d22000 rw-p f7d21000 00:00 0 f7d22000-f7e6e000 r-xp 00000000 fd:00 400450 /emul/ia32-linux/lib/libc-2.7.so f7e6e000-f7e6f000 r--p 0014c000 fd:00 400450 /emul/ia32-linux/lib/libc-2.7.so f7e6f000-f7e71000 rw-p 0014d000 fd:00 400450 /emul/ia32-linux/lib/libc-2.7.so f7e71000-f7e74000 rw-p f7e71000 00:00 0 f7e74000-f7e80000 r-xp 00000000 fd:00 49158 /emul/ia32-linux/usr/lib/libgcc_s.so.1 f7e80000-f7e81000 rw-p 0000b000 fd:00 49158 /emul/ia32-linux/usr/lib/libgcc_s.so.1 f7e81000-f7ea3000 r-xp 00000000 fd:00 400454 /emul/ia32-linux/lib/libm-2.7.so f7ea3000-f7ea5000 rw-p 00022000 fd:00 400454 /emul/ia32-linux/lib/libm-2.7.so f7ea5000-f7ea6000 rw-p f7ea5000 00:00 0 f7ea6000-f7f89000 r-xp 00000000 fd:00 49161 /emul/ia32-linux/usr/lib/libstdc++.so.6.0.10 f7f89000-f7f8c000 r--p 000e2000 fd:00 49161 /emul/ia32-linux/usr/lib/libstdc++.so.6.0.10 f7f8c000-f7f8e000 rw-p 000e5000 fd:00 49161 /emul/ia32-linux/usr/lib/libstdc++.so.6.0.10 f7f8e000-f7f94000 rw-p f7f8e000 00:00 0 f7f94000-f7f96000 r-xp 00000000 fd:00 400453 /emul/ia32-linux/lib/libdl-2.7.so f7f96000-f7f98000 rw-p 00001000 fd:00 400453 /emul/ia32-linux/lib/libdl-2.7.so f7f98000-f7fab000 r-xp 00000000 fd:00 400446 /emul/ia32-linux/lib/libpthread-2.7.so f7fab000-f7fad000 rw-p 00012000 fd:00 400446 /emul/ia32-linux/lib/libpthread-2.7.so f7fad000-f7faf000 rw-p f7fad000 00:00 0 f7fb2000-f7fb3000 rw-p f7fb2000 00:00 0 f7fb3000-f7fb4000 ---p f7fb3000 00:00 0 f7fb4000-f7fb7000 rw-p f7fb4000 00:00 0 f7fb7000-f7fb9000 rw-s 00000000 fd:00 117 /var/lib/boinc-client/slots/1/boinc_mmap_file f7fb9000-f7fbc000 rw-p f7fb9000 00:00 0 f7fbc000-f7fbd000 r-xp f7fbc000 00:00 0 [vdso] f7fbd000-f7fda000 r-xp 00000000 fd:00 400447 /emul/ia32-linux/lib/ld-2.7.so f7fda000-f7fdc000 rw-p 0001c000 fd:00 400447 /emul/ia32-linux/lib/ld-2.7.so ffd6d000-ffddc000 rw-p 7ffffff90000 00:00 0 [stack] SIGABRT: abort called Stack trace (17 frames): hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7fbc400] /lib32/libc.so.6(gsignal+0x3f)[0xf7d4cd7f] /lib32/libc.so.6(abort+0x188)[0xf7d4e788] /lib32/libc.so.6[0xf7d896f5] /lib32/libc.so.6[0xf7d8f10c] /lib32/libc.so.6(cfree+0x85)[0xf7d91145] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7f5e6b1] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7f5e70d] hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe5)[0xf7d3842d] hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Nov 2011 14:57:54	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	259,200	472,465	1.8228
05 Nov 2011 00:43:58	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	233,280	428,650	1.8375
04 Nov 2011 12:45:39	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	207,360	394,273	1.9014
04 Nov 2011 00:17:22	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	181,440	346,328	1.9088
03 Nov 2011 10:07:14	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	155,520	305,915	1.9670
02 Nov 2011 22:27:32	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	129,600	262,258	2.0236
01 Nov 2011 22:52:28	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	103,680	211,227	2.0373
01 Nov 2011 04:01:00	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	77,760	164,766	2.1189
31 Oct 2011 19:46:10	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	51,840	111,072	2.1426
31 Oct 2011 19:24:37	1111270	13566850	hadcm3n_yeck_1900_40_007525591_4	25,920	56,736	2.1889