climateprediction.net home page
Task 15793557

Task 15793557

Name hadcm3n_3889_1940_40_008265135_4
Workunit 8420259
Created 23 May 2013, 1:09:01 UTC
Sent 23 May 2013, 1:09:34 UTC
Report deadline 22 Aug 2013, 8:36:45 UTC
Received 30 May 2013, 22:19:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1282401
Run time 7 days 12 hours 49 min 23 sec
CPU time 7 days 8 hours 58 min 44 sec
Validate state Invalid
Credit 2,488.32
Device peak FLOPS 2.01 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.27</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
09:35:20 (7744): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
21:23:50 (13143): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:56:34 (19682): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
02:54:29 (52383): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:28:03 (10408): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:12:21 (14784): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:12:22 (14784): No heartbeat from core client for 30 sec - exiting
06:40:35 (37913): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:45:34 (38946): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:45:35 (38946): No heartbeat from core client for 30 sec - exiting
06:45:36 (38946): No heartbeat from core client for 30 sec - exiting
06:45:37 (38946): No heartbeat from core client for 30 sec - exiting
06:45:38 (38946): No heartbeat from core client for 30 sec - exiting
06:45:39 (38946): No heartbeat from core client for 30 sec - exiting
06:45:40 (38946): No heartbeat from core client for 30 sec - exiting
06:45:41 (38946): No heartbeat from core client for 30 sec - exiting
06:45:42 (38946): No heartbeat from core client for 30 sec - exiting
06:45:43 (38946): No heartbeat from core client for 30 sec - exiting
06:45:44 (38946): No heartbeat from core client for 30 sec - exiting
06:45:45 (38946): No heartbeat from core client for 30 sec - exiting
06:45:46 (38946): No heartbeat from core client for 30 sec - exiting
06:45:47 (38946): No heartbeat from core client for 30 sec - exiting
06:45:48 (38946): No heartbeat from core client for 30 sec - exiting
06:45:49 (38946): No heartbeat from core client for 30 sec - exiting
09:04:31 (39121): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:36:02 (40763): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:28 (41744): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:29 (41744): No heartbeat from core client for 30 sec - exiting
10:40:30 (41744): No heartbeat from core client for 30 sec - exiting
10:40:31 (41744): No heartbeat from core client for 30 sec - exiting
10:40:32 (41744): No heartbeat from core client for 30 sec - exiting
10:40:33 (41744): No heartbeat from core client for 30 sec - exiting
10:40:34 (41744): No heartbeat from core client for 30 sec - exiting
10:40:35 (41744): No heartbeat from core client for 30 sec - exiting
10:40:36 (41744): No heartbeat from core client for 30 sec - exiting
10:40:37 (41744): No heartbeat from core client for 30 sec - exiting
10:40:38 (41744): No heartbeat from core client for 30 sec - exiting
10:40:39 (41744): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
10:45:30 (41965): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:03:47 (42115): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:08:55 (44024): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:25 (44181): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:26 (44181): No heartbeat from core client for 30 sec - exiting
14:13:27 (44181): No heartbeat from core client for 30 sec - exiting
14:13:28 (44181): No heartbeat from core client for 30 sec - exiting
14:13:29 (44181): No heartbeat from core client for 30 sec - exiting
14:51:27 (44407): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:51:28 (44407): No heartbeat from core client for 30 sec - exiting
15:01:03 (44917): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:01:04 (44917): No heartbeat from core client for 30 sec - exiting
15:01:05 (44917): No heartbeat from core client for 30 sec - exiting
15:01:06 (44917): No heartbeat from core client for 30 sec - exiting
15:05:46 (45124): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:05:47 (45124): No heartbeat from core client for 30 sec - exiting
15:05:48 (45124): No heartbeat from core client for 30 sec - exiting
15:05:49 (45124): No heartbeat from core client for 30 sec - exiting
15:05:50 (45124): No heartbeat from core client for 30 sec - exiting
15:05:51 (45124): No heartbeat from core client for 30 sec - exiting
15:05:52 (45124): No heartbeat from core client for 30 sec - exiting
15:05:53 (45124): No heartbeat from core client for 30 sec - exiting
15:05:54 (45124): No heartbeat from core client for 30 sec - exiting
15:05:55 (45124): No heartbeat from core client for 30 sec - exiting
15:05:56 (45124): No heartbeat from core client for 30 sec - exiting
15:05:57 (45124): No heartbeat from core client for 30 sec - exiting
15:05:58 (45124): No heartbeat from core client for 30 sec - exiting
15:05:59 (45124): No heartbeat from core client for 30 sec - exiting
15:06:00 (45124): No heartbeat from core client for 30 sec - exiting
15:06:01 (45124): No heartbeat from core client for 30 sec - exiting
15:06:02 (45124): No heartbeat from core client for 30 sec - exiting
15:06:03 (45124): No heartbeat from core client for 30 sec - exiting
15:06:04 (45124): No heartbeat from core client for 30 sec - exiting
15:42:25 (45343): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:47:25 (45786): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:47:26 (45786): No heartbeat from core client for 30 sec - exiting
16:26:39 (45985): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:31:38 (46440): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:31:39 (46440): No heartbeat from core client for 30 sec - exiting
16:31:40 (46440): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
16:43:01 (46656): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:03:45 (46840): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:29:13 (47104): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:29:14 (47104): No heartbeat from core client for 30 sec - exiting
17:29:15 (47104): No heartbeat from core client for 30 sec - exiting
17:29:16 (47104): No heartbeat from core client for 30 sec - exiting
17:29:17 (47104): No heartbeat from core client for 30 sec - exiting
17:29:18 (47104): No heartbeat from core client for 30 sec - exiting
17:29:19 (47104): No heartbeat from core client for 30 sec - exiting
17:29:20 (47104): No heartbeat from core client for 30 sec - exiting
17:29:21 (47104): No heartbeat from core client for 30 sec - exiting
17:29:22 (47104): No heartbeat from core client for 30 sec - exiting
17:29:23 (47104): No heartbeat from core client for 30 sec - exiting
17:29:24 (47104): No heartbeat from core client for 30 sec - exiting
17:38:36 (47505): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:38:37 (47505): No heartbeat from core client for 30 sec - exiting
17:38:38 (47505): No heartbeat from core client for 30 sec - exiting
17:38:39 (47505): No heartbeat from core client for 30 sec - exiting
17:38:40 (47505): No heartbeat from core client for 30 sec - exiting
17:38:41 (47505): No heartbeat from core client for 30 sec - exiting
17:38:42 (47505): No heartbeat from core client for 30 sec - exiting
18:19:10 (47723): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:19:11 (47723): No heartbeat from core client for 30 sec - exiting
18:19:12 (47723): No heartbeat from core client for 30 sec - exiting
18:19:13 (47723): No heartbeat from core client for 30 sec - exiting
18:23:44 (48236): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:22 (48445): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:23 (48445): No heartbeat from core client for 30 sec - exiting
18:28:24 (48445): No heartbeat from core client for 30 sec - exiting
18:41:08 (48637): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:41:09 (48637): No heartbeat from core client for 30 sec - exiting
18:41:10 (48637): No heartbeat from core client for 30 sec - exiting
19:51:15 (48849): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:00 (49621): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:01 (49621): No heartbeat from core client for 30 sec - exiting
19:56:02 (49621): No heartbeat from core client for 30 sec - exiting
19:56:03 (49621): No heartbeat from core client for 30 sec - exiting
19:56:04 (49621): No heartbeat from core client for 30 sec - exiting
19:56:05 (49621): No heartbeat from core client for 30 sec - exiting
19:56:06 (49621): No heartbeat from core client for 30 sec - exiting
19:56:07 (49621): No heartbeat from core client for 30 sec - exiting
19:56:08 (49621): No heartbeat from core client for 30 sec - exiting
19:56:09 (49621): No heartbeat from core client for 30 sec - exiting
19:56:10 (49621): No heartbeat from core client for 30 sec - exiting
19:56:11 (49621): No heartbeat from core client for 30 sec - exiting
19:56:12 (49621): No heartbeat from core client for 30 sec - exiting
19:56:13 (49621): No heartbeat from core client for 30 sec - exiting
19:56:14 (49621): No heartbeat from core client for 30 sec - exiting
19:56:15 (49621): No heartbeat from core client for 30 sec - exiting
19:56:16 (49621): No heartbeat from core client for 30 sec - exiting
19:56:17 (49621): No heartbeat from core client for 30 sec - exiting
19:56:18 (49621): No heartbeat from core client for 30 sec - exiting
19:56:19 (49621): No heartbeat from core client for 30 sec - exiting
19:56:20 (49621): No heartbeat from core client for 30 sec - exiting
19:56:21 (49621): No heartbeat from core client for 30 sec - exiting
19:56:22 (49621): No heartbeat from core client for 30 sec - exiting
19:56:23 (49621): No heartbeat from core client for 30 sec - exiting
19:56:24 (49621): No heartbeat from core client for 30 sec - exiting
19:56:25 (49621): No heartbeat from core client for 30 sec - exiting
20:00:34 (49807): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:01:05 (49807): No heartbeat from core client for 30 sec - exiting
21:12:14 (50003): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:03 (50770): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:04 (50770): No heartbeat from core client for 30 sec - exiting
21:17:05 (50770): No heartbeat from core client for 30 sec - exiting
21:17:06 (50770): No heartbeat from core client for 30 sec - exiting
21:17:07 (50770): No heartbeat from core client for 30 sec - exiting
21:17:08 (50770): No heartbeat from core client for 30 sec - exiting
21:17:09 (50770): No heartbeat from core client for 30 sec - exiting
21:17:10 (50770): No heartbeat from core client for 30 sec - exiting
21:17:11 (50770): No heartbeat from core client for 30 sec - exiting
21:17:12 (50770): No heartbeat from core client for 30 sec - exiting
21:17:13 (50770): No heartbeat from core client for 30 sec - exiting
21:17:14 (50770): No heartbeat from core client for 30 sec - exiting
21:17:15 (50770): No heartbeat from core client for 30 sec - exiting
21:17:16 (50770): No heartbeat from core client for 30 sec - exiting
21:17:17 (50770): No heartbeat from core client for 30 sec - exiting
21:17:18 (50770): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf77c6400]
[0xf77c6425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75e31df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75e6825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75ce4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf777e400]
[0xf777e425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf759b1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf759e825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75864d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf76e6400]
[0xf76e6425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75031df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7506825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf74ee4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7719400]
[0xf7719425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75361df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7539825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75214d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7741400]
[0xf7741425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf755e1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7561825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75494d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf770c400]
[0xf770c425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75291df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf752c825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75144d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50982, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
30 May 2013 05:58:04 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 207,360 594,990 2.8694
29 May 2013 06:52:21 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 181,440 514,380 2.8350
28 May 2013 07:53:26 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 155,520 437,711 2.8145
27 May 2013 12:10:07 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 129,600 367,557 2.8361
26 May 2013 16:02:24 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 103,680 298,374 2.8778
25 May 2013 18:04:33 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 77,760 224,897 2.8922
24 May 2013 20:36:55 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 51,840 149,685 2.8874
23 May 2013 22:39:21 1282401 15793557 hadcm3n_3889_1940_40_008265135_4 25,920 71,314 2.7513


©2024 climateprediction.net