Task 13409476

Name	hadcm3n_u6mu_1980_40_007459434_1
Workunit	7656937
Created	22 Sep 2011, 19:16:27 UTC
Sent	24 Sep 2011, 23:52:56 UTC
Report deadline	25 Dec 2011, 7:20:07 UTC
Received	1 Oct 2011, 19:07:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1066269
Run time	4 days 20 hours 17 min 31 sec
CPU time	3 days 0 hours 42 min 6 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	2.94 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 03:49:19 (13014): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:20 (13014): No heartbeat from core client for 30 sec - exiting 03:49:21 (13014): No heartbeat from core client for 30 sec - exiting 03:49:22 (13014): No heartbeat from core client for 30 sec - exiting 03:49:23 (13014): No heartbeat from core client for 30 sec - exiting 03:49:24 (13014): No heartbeat from core client for 30 sec - exiting 03:49:25 (13014): No heartbeat from core client for 30 sec - exiting 03:49:26 (13014): No heartbeat from core client for 30 sec - exiting 03:49:27 (13014): No heartbeat from core client for 30 sec - exiting 03:49:28 (13014): No heartbeat from core client for 30 sec - exiting 03:49:29 (13014): No heartbeat from core client for 30 sec - exiting 03:49:30 (13014): No heartbeat from core client for 30 sec - exiting 03:49:36 (13014): No heartbeat from core client for 30 sec - exiting 03:49:37 (13014): No heartbeat from core client for 30 sec - exiting 03:49:38 (13014): No heartbeat from core client for 30 sec - exiting 03:49:39 (13014): No heartbeat from core client for 30 sec - exiting 03:49:40 (13014): No heartbeat from core client for 30 sec - exiting 03:49:41 (13014): No heartbeat from core client for 30 sec - exiting 03:49:42 (13014): No heartbeat from core client for 30 sec - exiting 03:49:43 (13014): No heartbeat from core client for 30 sec - exiting 04:25:06 (13524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:25:07 (13524): No heartbeat from core client for 30 sec - exiting 04:25:08 (13524): No heartbeat from core client for 30 sec - exiting 04:25:09 (13524): No heartbeat from core client for 30 sec - exiting 04:25:10 (13524): No heartbeat from core client for 30 sec - exiting 04:25:11 (13524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 21:27:25 (30201): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:27:26 (30201): No heartbeat from core client for 30 sec - exiting 21:27:27 (30201): No heartbeat from core client for 30 sec - exiting 21:27:28 (30201): No heartbeat from core client for 30 sec - exiting 21:27:29 (30201): No heartbeat from core client for 30 sec - exiting 21:27:30 (30201): No heartbeat from core client for 30 sec - exiting 23:30:54 (32326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:30:55 (32326): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 01:03:39 (326): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:03:41 (326): No heartbeat from core client for 30 sec - exiting 01:03:42 (326): No heartbeat from core client for 30 sec - exiting 01:03:43 (326): No heartbeat from core client for 30 sec - exiting 01:03:44 (326): No heartbeat from core client for 30 sec - exiting 01:03:45 (326): No heartbeat from core client for 30 sec - exiting 01:03:46 (326): No heartbeat from core client for 30 sec - exiting 01:03:47 (326): No heartbeat from core client for 30 sec - exiting 01:03:54 (326): No heartbeat from core client for 30 sec - exiting 01:03:55 (326): No heartbeat from core client for 30 sec - exiting 01:13:13 (721): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:14 (721): No heartbeat from core client for 30 sec - exiting 01:13:15 (721): No heartbeat from core client for 30 sec - exiting 01:13:16 (721): No heartbeat from core client for 30 sec - exiting 01:13:17 (721): No heartbeat from core client for 30 sec - exiting 01:13:18 (721): No heartbeat from core client for 30 sec - exiting 01:13:19 (721): No heartbeat from core client for 30 sec - exiting 01:13:28 (721): No heartbeat from core client for 30 sec - exiting 01:13:36 (721): No heartbeat from core client for 30 sec - exiting 01:13:38 (721): No heartbeat from core client for 30 sec - exiting 01:13:39 (721): No heartbeat from core client for 30 sec - exiting 01:13:40 (721): No heartbeat from core client for 30 sec - exiting 01:13:41 (721): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 07:14:06 (2011): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:14:07 (2011): No heartbeat from core client for 30 sec - exiting 07:14:08 (2011): No heartbeat from core client for 30 sec - exiting 07:14:17 (2011): No heartbeat from core client for 30 sec - exiting 07:14:25 (2011): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:18:26 (2259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:18:27 (2259): No heartbeat from core client for 30 sec - exiting 09:18:28 (2259): No heartbeat from core client for 30 sec - exiting 09:18:29 (2259): No heartbeat from core client for 30 sec - exiting 09:18:30 (2259): No heartbeat from core client for 30 sec - exiting 09:18:31 (2259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:35:11 (15645): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:22:33 (27270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:34 (27270): No heartbeat from core client for 30 sec - exiting 18:22:35 (27270): No heartbeat from core client for 30 sec - exiting 18:22:36 (27270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:06:17 (5965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:06:18 (5965): No heartbeat from core client for 30 sec - exiting 01:06:19 (5965): No heartbeat from core client for 30 sec - exiting 01:06:20 (5965): No heartbeat from core client for 30 sec - exiting 01:06:21 (5965): No heartbeat from core client for 30 sec - exiting 01:06:29 (5965): No heartbeat from core client for 30 sec - exiting 01:06:30 (5965): No heartbeat from core client for 30 sec - exiting 01:06:31 (5965): No heartbeat from core client for 30 sec - exiting 01:06:32 (5965): No heartbeat from core client for 30 sec - exiting 01:06:33 (5965): No heartbeat from core client for 30 sec - exiting 01:06:39 (5965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:29:26 (25852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:29:27 (25852): No heartbeat from core client for 30 sec - exiting 10:29:28 (25852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:48:08 (26636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:48:09 (26636): No heartbeat from core client for 30 sec - exiting 15:48:10 (26636): No heartbeat from core client for 30 sec - exiting 15:48:11 (26636): No heartbeat from core client for 30 sec - exiting 15:48:12 (26636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 04:56:21 (13505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:56:23 (13505): No heartbeat from core client for 30 sec - exiting 04:56:24 (13505): No heartbeat from core client for 30 sec - exiting 04:56:25 (13505): No heartbeat from core client for 30 sec - exiting 04:56:26 (13505): No heartbeat from core client for 30 sec - exiting 04:56:27 (13505): No heartbeat from core client for 30 sec - exiting 04:56:28 (13505): No heartbeat from core client for 30 sec - exiting 05:07:54 (15096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:55 (15096): No heartbeat from core client for 30 sec - exiting 05:08:04 (15096): No heartbeat from core client for 30 sec - exiting 05:08:14 (15096): No heartbeat from core client for 30 sec - exiting 05:08:16 (15096): No heartbeat from core client for 30 sec - exiting 05:08:17 (15096): No heartbeat from core client for 30 sec - exiting 05:08:18 (15096): No heartbeat from core client for 30 sec - exiting 05:08:19 (15096): No heartbeat from core client for 30 sec - exiting 05:08:25 (15096): No heartbeat from core client for 30 sec - exiting 05:08:26 (15096): No heartbeat from core client for 30 sec - exiting 05:08:27 (15096): No heartbeat from core client for 30 sec - exiting 05:08:28 (15096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:33:14 (16055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:15 (16055): No heartbeat from core client for 30 sec - exiting 10:33:16 (16055): No heartbeat from core client for 30 sec - exiting 10:33:17 (16055): No heartbeat from core client for 30 sec - exiting 10:33:18 (16055): No heartbeat from core client for 30 sec - exiting 10:33:19 (16055): No heartbeat from core client for 30 sec - exiting 10:33:20 (16055): No heartbeat from core client for 30 sec - exiting 10:33:21 (16055): No heartbeat from core client for 30 sec - exiting 10:33:29 (16055): No heartbeat from core client for 30 sec - exiting 10:33:30 (16055): No heartbeat from core client for 30 sec - exiting 10:33:31 (16055): No heartbeat from core client for 30 sec - exiting 10:33:32 (16055): No heartbeat from core client for 30 sec - exiting 10:33:33 (16055): No heartbeat from core client for 30 sec - exiting 10:33:34 (16055): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 10:41:22 (16524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:06:27 (16566): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:06:28 (16566): No heartbeat from core client for 30 sec - exiting 11:06:29 (16566): No heartbeat from core client for 30 sec - exiting 13:44:58 (30382): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:59 (30382): No heartbeat from core client for 30 sec - exiting 13:45:00 (30382): No heartbeat from core client for 30 sec - exiting 09:00:35 (29363): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:36 (29363): No heartbeat from core client for 30 sec - exiting 09:00:37 (29363): No heartbeat from core client for 30 sec - exiting 09:00:38 (29363): No heartbeat from core client for 30 sec - exiting 09:00:39 (29363): No heartbeat from core client for 30 sec - exiting 09:00:40 (29363): No heartbeat from core client for 30 sec - exiting 09:00:41 (29363): No heartbeat from core client for 30 sec - exiting 09:00:42 (29363): No heartbeat from core client for 30 sec - exiting 09:13:41 (21554): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:13:42 (21554): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 20:01:46 (11969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:47 (11969): No heartbeat from core client for 30 sec - exiting 20:01:48 (11969): No heartbeat from core client for 30 sec - exiting 20:01:49 (11969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:22:07 (15975): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:22:09 (15975): No heartbeat from core client for 30 sec - exiting 07:22:10 (15975): No heartbeat from core client for 30 sec - exiting 07:22:11 (15975): No heartbeat from core client for 30 sec - exiting 07:22:12 (15975): No heartbeat from core client for 30 sec - exiting 07:22:13 (15975): No heartbeat from core client for 30 sec - exiting 07:22:14 (15975): No heartbeat from core client for 30 sec - exiting 07:22:15 (15975): No heartbeat from core client for 30 sec - exiting 14:07:32 (16048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:06 (20585): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:07 (20585): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 12:45:40 (28924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:41 (28924): No heartbeat from core client for 30 sec - exiting 12:45:42 (28924): No heartbeat from core client for 30 sec - exiting 12:45:43 (28924): No heartbeat from core client for 30 sec - exiting 12:45:44 (28924): No heartbeat from core client for 30 sec - exiting 12:45:45 (28924): No heartbeat from core client for 30 sec - exiting 12:45:46 (28924): No heartbeat from core client for 30 sec - exiting 12:45:47 (28924): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 16:48:31 (32684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:48:32 (32684): No heartbeat from core client for 30 sec - exiting 16:48:33 (32684): No heartbeat from core client for 30 sec - exiting 16:48:34 (32684): No heartbeat from core client for 30 sec - exiting 16:48:35 (32684): No heartbeat from core client for 30 sec - exiting 16:48:36 (32684): No heartbeat from core client for 30 sec - exiting 16:48:37 (32684): No heartbeat from core client for 30 sec - exiting 16:48:38 (32684): No heartbeat from core client for 30 sec - exiting 16:48:39 (32684): No heartbeat from core client for 30 sec - exiting 16:48:40 (32684): No heartbeat from core client for 30 sec - exiting 16:48:47 (32684): No heartbeat from core client for 30 sec - exiting 16:48:54 (32684): No heartbeat from core client for 30 sec - exiting 16:48:55 (32684): No heartbeat from core client for 30 sec - exiting 20:00:23 (32741): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:29 (32741): No heartbeat from core client for 30 sec - exiting 20:04:23 (28827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:04:24 (28827): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf779c400] [0xf779c430] /lib32/libc.so.6(gsignal+0x50)[0xf75bf710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf75a7e4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7748400] [0xf7748430] /lib32/libc.so.6(gsignal+0x50)[0xf756b710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf7553e4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77a1400] [0xf77a1430] /lib32/libc.so.6(gsignal+0x50)[0xf75c4710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf75ace4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7714400] [0xf7714430] /lib32/libc.so.6(gsignal+0x50)[0xf7537710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf751fe4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77c0400] [0xf77c0430] /lib32/libc.so.6(gsignal+0x50)[0xf75e3710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf75cbe4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (8 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf76f2400] [0xf76f2430] /lib32/libc.so.6(gsignal+0x50)[0xf7515710] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xff)[0xf74fde4f] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=15731, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Oct 2011 06:36:24	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	155,520	239,011	1.5369
30 Sep 2011 08:51:12	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	129,600	198,511	1.5317
29 Sep 2011 08:17:44	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	103,680	157,985	1.5238
28 Sep 2011 15:36:04	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	77,760	118,933	1.5295
27 Sep 2011 21:35:54	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	51,840	79,557	1.5347
26 Sep 2011 16:19:49	1066269	13409476	hadcm3n_u6mu_1980_40_007459434_1	25,920	39,520	1.5247