Task 13462093

Name	hadcm3n_t1nq_1940_40_007435323_2
Workunit	7632826
Created	5 Oct 2011, 7:16:43 UTC
Sent	5 Oct 2011, 7:16:44 UTC
Report deadline	4 Jan 2012, 14:43:55 UTC
Received	24 Nov 2011, 14:57:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1147236
Run time	7 days 7 hours 8 min 14 sec
CPU time	7 days 4 hours 26 min 24 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:28 (1702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:31:38 (2572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:27 (2852): No heartbeat from core client for 30 sec - exiting 08:16:28 (2852): No heartbeat from core client for 30 sec - exiting 08:16:29 (2852): No heartbeat from core client for 30 sec - exiting 08:16:30 (2852): No heartbeat from core client for 30 sec - exiting 08:16:31 (2852): No heartbeat from core client for 30 sec - exiting 08:16:32 (2852): No heartbeat from core client for 30 sec - exiting 08:16:33 (2852): No heartbeat from core client for 30 sec - exiting 08:16:34 (2852): No heartbeat from core client for 30 sec - exiting 08:16:35 (2852): No heartbeat from core client for 30 sec - exiting 08:16:36 (2852): No heartbeat from core client for 30 sec - exiting 08:16:37 (2852): No heartbeat from core client for 30 sec - exiting 08:16:38 (2852): No heartbeat from core client for 30 sec - exiting 08:16:39 (2852): No heartbeat from core client for 30 sec - exiting 08:16:40 (2852): No heartbeat from core client for 30 sec - exiting 08:16:41 (2852): No heartbeat from core client for 30 sec - exiting 08:16:42 (2852): No heartbeat from core client for 30 sec - exiting 08:16:43 (2852): No heartbeat from core client for 30 sec - exiting 08:16:44 (2852): No heartbeat from core client for 30 sec - exiting 08:16:45 (2852): No heartbeat from core client for 30 sec - exiting 08:16:46 (2852): No heartbeat from core client for 30 sec - exiting 08:16:47 (2852): No heartbeat from core client for 30 sec - exiting 08:16:48 (2852): No heartbeat from core client for 30 sec - exiting 08:16:49 (2852): No heartbeat from core client for 30 sec - exiting 08:16:50 (2852): No heartbeat from core client for 30 sec - exiting 08:16:51 (2852): No heartbeat from core client for 30 sec - exiting 08:16:52 (2852): No heartbeat from core client for 30 sec - exiting 08:16:53 (2852): No heartbeat from core client for 30 sec - exiting 08:16:54 (2852): No heartbeat from core client for 30 sec - exiting 08:16:55 (2852): No heartbeat from core client for 30 sec - exiting 08:16:56 (2852): No heartbeat from core client for 30 sec - exiting 08:16:57 (2852): No heartbeat from core client for 30 sec - exiting 08:16:58 (2852): No heartbeat from core client for 30 sec - exiting 08:16:59 (2852): No heartbeat from core client for 30 sec - exiting 08:17:00 (2852): No heartbeat from core client for 30 sec - exiting 08:17:01 (2852): No heartbeat from core client for 30 sec - exiting 08:17:02 (2852): No heartbeat from core client for 30 sec - exiting 08:17:03 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:36:05 (7689): Can't acquire lockfile (-154) - waiting 35s Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6869, selfPID=6869, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1725, selfPID=1725, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:57:03 (1570): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:05 (1570): No heartbeat from core client for 30 sec - exiting 08:57:06 (1570): No heartbeat from core client for 30 sec - exiting 08:57:07 (1570): No heartbeat from core client for 30 sec - exiting 08:57:08 (1570): No heartbeat from core client for 30 sec - exiting 08:57:09 (1570): No heartbeat from core client for 30 sec - exiting 08:57:10 (1570): No heartbeat from core client for 30 sec - exiting 08:57:11 (1570): No heartbeat from core client for 30 sec - exiting 08:57:12 (1570): No heartbeat from core client for 30 sec - exiting 08:57:13 (1570): No heartbeat from core client for 30 sec - exiting 08:57:14 (1570): No heartbeat from core client for 30 sec - exiting 08:57:15 (1570): No heartbeat from core client for 30 sec - exiting 08:57:16 (1570): No heartbeat from core client for 30 sec - exiting 08:57:17 (1570): No heartbeat from core client for 30 sec - exiting 08:57:18 (1570): No heartbeat from core client for 30 sec - exiting 08:57:19 (1570): No heartbeat from core client for 30 sec - exiting 08:57:20 (1570): No heartbeat from core client for 30 sec - exiting 08:57:21 (1570): No heartbeat from core client for 30 sec - exiting 08:57:22 (1570): No heartbeat from core client for 30 sec - exiting 08:57:23 (1570): No heartbeat from core client for 30 sec - exiting 08:57:24 (1570): No heartbeat from core client for 30 sec - exiting 08:57:25 (1570): No heartbeat from core client for 30 sec - exiting 08:57:26 (1570): No heartbeat from core client for 30 sec - exiting 08:57:27 (1570): No heartbeat from core client for 30 sec - exiting 08:57:28 (1570): No heartbeat from core client for 30 sec - exiting 08:57:29 (1570): No heartbeat from core client for 30 sec - exiting 08:57:30 (1570): No heartbeat from core client for 30 sec - exiting 08:57:31 (1570): No heartbeat from core client for 30 sec - exiting 08:57:32 (1570): No heartbeat from core client for 30 sec - exiting 08:57:33 (1570): No heartbeat from core client for 30 sec - exiting 08:57:34 (1570): No heartbeat from core client for 30 sec - exiting 08:57:35 (1570): No heartbeat from core client for 30 sec - exiting 08:57:36 (1570): No heartbeat from core client for 30 sec - exiting 08:57:37 (1570): No heartbeat from core client for 30 sec - exiting 08:57:38 (1570): No heartbeat from core client for 30 sec - exiting 08:57:39 (1570): No heartbeat from core client for 30 sec - exiting 08:57:40 (1570): No heartbeat from core client for 30 sec - exiting 08:57:41 (1570): No heartbeat from core client for 30 sec - exiting 08:57:42 (1570): No heartbeat from core client for 30 sec - exiting 08:57:43 (1570): No heartbeat from core client for 30 sec - exiting 08:57:44 (1570): No heartbeat from core client for 30 sec - exiting 08:57:45 (1570): No heartbeat from core client for 30 sec - exiting 08:57:46 (1570): No heartbeat from core client for 30 sec - exiting 08:57:47 (1570): No heartbeat from core client for 30 sec - exiting 08:57:48 (1570): No heartbeat from core client for 30 sec - exiting 08:57:49 (1570): No heartbeat from core client for 30 sec - exiting 08:57:50 (1570): No heartbeat from core client for 30 sec - exiting 08:57:51 (1570): No heartbeat from core client for 30 sec - exiting 08:57:52 (1570): No heartbeat from core client for 30 sec - exiting 08:57:53 (1570): No heartbeat from core client for 30 sec - exiting 08:57:54 (1570): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09f09fe8 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6b511)[0xf7502511] /lib32/libc.so.6(+0x6ce1b)[0xf7503e1b] /lib32/libc.so.6(cfree+0x6d)[0xf7506f8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76e0841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76e089d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74ade37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:05 1707825 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:05 1707825 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09eb4000-09f1a000 rw-p 00000000 00:00 0 [heap] f6f00000-f6f21000 rw-p 00000000 00:00 0 f6f21000-f7000000 ---p 00000000 00:00 0 f7019000-f7494000 rw-s 00000000 08:05 1714527 /var/lib/boinc-client/slots/5/136470 f7494000-f7497000 rw-p 00000000 00:00 0 f7497000-f75ed000 r-xp 00000000 08:05 1700614 /lib32/libc-2.13.so f75ed000-f75ee000 ---p 00156000 08:05 1700614 /lib32/libc-2.13.so f75ee000-f75f0000 r--p 00156000 08:05 1700614 /lib32/libc-2.13.so f75f0000-f75f1000 rw-p 00158000 08:05 1700614 /lib32/libc-2.13.so f75f1000-f75f4000 rw-p 00000000 00:00 0 f75f4000-f760e000 r-xp 00000000 08:05 532484 /usr/lib32/libgcc_s.so.1 f760e000-f760f000 r--p 00019000 08:05 532484 /usr/lib32/libgcc_s.so.1 f760f000-f7610000 rw-p 0001a000 08:05 532484 /usr/lib32/libgcc_s.so.1 f7610000-f7634000 r-xp 00000000 08:05 1700618 /lib32/libm-2.13.so f7634000-f7635000 r--p 00023000 08:05 1700618 /lib32/libm-2.13.so f7635000-f7636000 rw-p 00024000 08:05 1700618 /lib32/libm-2.13.so f7636000-f7715000 r-xp 00000000 08:05 532491 /usr/lib32/libstdc++.so.6.0.14 f7715000-f7719000 r--p 000de000 08:05 532491 /usr/lib32/libstdc++.so.6.0.14 f7719000-f771a000 rw-p 000e2000 08:05 532491 /usr/lib32/libstdc++.so.6.0.14 f771a000-f7721000 rw-p 00000000 00:00 0 f7721000-f7723000 r-xp 00000000 08:05 1700617 /lib32/libdl-2.13.so f7723000-f7724000 r--p 00001000 08:05 1700617 /lib32/libdl-2.13.so f7724000-f7725000 rw-p 00002000 08:05 1700617 /lib32/libdl-2.13.so f7725000-f773a000 r-xp 00000000 08:05 1700628 /lib32/libpthread-2.13.so f773a000-f773b000 r--p 00014000 08:05 1700628 /lib32/libpthread-2.13.so f773b000-f773c000 rw-p 00015000 08:05 1700628 /lib32/libpthread-2.13.so f773c000-f773f000 rw-p 00000000 00:00 0 f775c000-f775d000 rw-p 00000000 00:00 0 f775d000-f775e000 ---p 00000000 00:00 0 f775e000-f7761000 rw-p 00000000 00:00 0 f7761000-f7763000 rw-s 00000000 08:05 1714515 /var/lib/boinc-client/slots/5/boinc_mmap_file f7763000-f7764000 rw-p 00000000 00:00 0 f7764000-f7765000 r-xp 00000000 00:00 0 [vdso] f7765000-f7781000 r-xp 00000000 08:05 1700611 /lib32/ld-2.13.so f7781000-f7782000 r--p 0001b000 08:05 1700611 /lib32/ld-2.13.so f7782000-f7783000 rw-p 0001c000 08:05 1700611 /lib32/ld-2.13.so fff49000-fffb8000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7764400] [0xf7764430] /lib32/libc.so.6(gsignal+0x51)[0xf74c1ea1] /lib32/libc.so.6(abort+0x17e)[0xf74c52ce] /lib32/libc.so.6(+0x61277)[0xf74f8277] /lib32/libc.so.6(+0x6b511)[0xf7502511] /lib32/libc.so.6(+0x6ce1b)[0xf7503e1b] /lib32/libc.so.6(cfree+0x6d)[0xf7506f8d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf76e0841] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf76e089d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe7)[0xf74ade37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Nov 2011 14:58:18	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	518,400	620,781	1.1975
18 Nov 2011 12:10:06	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	492,480	590,705	1.1994
16 Nov 2011 08:29:49	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	466,560	556,806	1.1934
09 Nov 2011 10:48:09	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	440,640	527,979	1.1982
08 Nov 2011 12:11:44	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	414,720	498,718	1.2025
07 Nov 2011 11:29:00	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	388,800	468,080	1.2039
03 Nov 2011 15:45:57	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	362,880	433,473	1.1945
02 Nov 2011 13:09:47	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	336,960	396,561	1.1769
31 Oct 2011 17:35:54	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	311,040	361,919	1.1636
31 Oct 2011 15:48:51	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	285,120	327,823	1.1498
31 Oct 2011 15:41:09	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	259,200	301,541	1.1634
31 Oct 2011 15:41:09	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	233,280	274,876	1.1783
31 Oct 2011 15:41:09	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	207,360	245,380	1.1834
19 Oct 2011 06:39:04	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	181,440	211,514	1.1658
18 Oct 2011 07:45:19	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	155,520	182,477	1.1733
17 Oct 2011 07:55:10	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	129,600	152,813	1.1791
13 Oct 2011 08:03:51	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	103,680	124,216	1.1981
11 Oct 2011 15:57:51	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	77,760	94,476	1.2150
10 Oct 2011 13:40:26	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	51,840	63,552	1.2259
07 Oct 2011 08:00:00	1147236	13462093	hadcm3n_t1nq_1940_40_007435323_2	25,920	29,204	1.1267