Task 17354074

Name	hadcm3n_xaan_1940_40_009149637_0
Workunit	9279973
Created	6 Nov 2014, 14:35:20 UTC
Sent	8 Nov 2014, 12:58:59 UTC
Report deadline	7 Feb 2015, 20:26:10 UTC
Received	10 Dec 2014, 18:07:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	127 (0x0000007F) Unknown error code
Computer ID	1408374
Run time	3 days 3 hours 8 min 25 sec
CPU time	2 days 19 hours 21 min 6 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	4.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 127 (0x7f, -129) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: Interrupted system call forrtl: severe (38): error during write, unit 6, file /media/WD600_5_26GB/BOINC/projects/climateprediction.net/hadcm3n_xaan_1940_40_009149637/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 080BBB15 Unknown Unknown Unknown hadcm3n_um_6.07_i 0807F9A8 Unknown Unknown Unknown hadcm3n_um_6.07_i 0837E9F4 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839982E Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F7556A83 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8314, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:10:31 (8257): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08757458 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 12:17:47 (30408): No heartbeat from core client for 30 sec - exiting 12:17:48 (30408): No heartbeat from core client for 30 sec - exiting 12:17:49 (30408): No heartbeat from core client for 30 sec - exiting 12:17:50 (30408): No heartbeat from core client for 30 sec - exiting 12:17:51 (30408): No heartbeat from core client for 30 sec - exiting 12:17:52 (30408): No heartbeat from core client for 30 sec - exiting 12:17:53 (30408): No heartbeat from core client for 30 sec - exiting 12:17:54 (30408): No heartbeat from core client for 30 sec - exiting 12:17:55 (30408): No heartbeat from core client for 30 sec - exiting 12:17:56 (30408): No heartbeat from core client for 30 sec - exiting 12:17:57 (30408): No heartbeat from core client for 30 sec - exiting 12:17:58 (30408): No heartbeat from core client for 30 sec - exiting 12:17:59 (30408): No heartbeat from core client for 30 sec - exiting 12:18:00 (30408): No heartbeat from core client for 30 sec - exiting 12:18:01 (30408): No heartbeat from core client for 30 sec - exiting 12:18:02 (30408): No heartbeat from core client for 30 sec - exiting 12:18:03 (30408): No heartbeat from core client for 30 sec - exiting 12:18:04 (30408): No heartbeat from core client for 30 sec - exiting 12:18:05 (30408): No heartbeat from core client for 30 sec - exiting 12:18:06 (30408): No heartbeat from core client for 30 sec - exiting 12:18:07 (30408): No heartbeat from core client for 30 sec - exiting 12:18:08 (30408): No heartbeat from core client for 30 sec - exiting 12:18:09 (30408): No heartbeat from core client for 30 sec - exiting 12:18:10 (30408): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x081b0ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 15:50:55 (7041): No heartbeat from core client for 30 sec - exiting 15:50:56 (7041): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09ad9ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09f9fab0 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x085c8ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x086e5ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x089a9ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x090adad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a151ad8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Nov 2014 07:19:26	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	259,200	225,799	0.8711
20 Nov 2014 20:19:04	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	233,280	203,065	0.8705
20 Nov 2014 01:30:18	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	207,360	180,069	0.8684
19 Nov 2014 17:24:37	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	181,440	155,157	0.8551
13 Nov 2014 15:30:10	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	155,520	132,182	0.8499
10 Nov 2014 07:49:23	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	129,600	113,425	0.8752
09 Nov 2014 21:22:31	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	103,680	90,777	0.8755
09 Nov 2014 09:02:04	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	77,760	68,171	0.8767
09 Nov 2014 02:23:40	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	51,840	45,468	0.8771
08 Nov 2014 19:34:43	1241544	17354074	hadcm3n_xaan_1940_40_009149637_0	25,920	22,833	0.8809