climateprediction.net home page
Task 15782964

Task 15782964

Name hadcm3n_4icu_1940_40_008311853_2
Workunit 8462988
Created 14 May 2013, 10:30:42 UTC
Sent 14 May 2013, 10:31:49 UTC
Report deadline 13 Aug 2013, 17:59:00 UTC
Received 30 May 2013, 22:19:47 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1282401
Run time 16 days 2 hours 20 min 38 sec
CPU time 15 days 18 hours 32 min 14 sec
Validate state Invalid
Credit 5,909.76
Device peak FLOPS 2.01 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.27</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
15:42:45 (22208): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
17:25:14 (2329): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:25:16 (2329): No heartbeat from core client for 30 sec - exiting
17:25:17 (2329): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
09:35:20 (2961): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
09:35:21 (2961): No heartbeat from core client for 30 sec - exiting
09:35:22 (2961): No heartbeat from core client for 30 sec - exiting
09:35:23 (2961): No heartbeat from core client for 30 sec - exiting
09:35:24 (2961): No heartbeat from core client for 30 sec - exiting
09:35:25 (2961): No heartbeat from core client for 30 sec - exiting
09:35:26 (2961): No heartbeat from core client for 30 sec - exiting
09:35:27 (2961): No heartbeat from core client for 30 sec - exiting
09:35:28 (2961): No heartbeat from core client for 30 sec - exiting
09:35:29 (2961): No heartbeat from core client for 30 sec - exiting
09:35:30 (2961): No heartbeat from core client for 30 sec - exiting
09:35:31 (2961): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
02:54:28 (13119): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:40:35 (10373): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:40:36 (10373): No heartbeat from core client for 30 sec - exiting
06:40:37 (10373): No heartbeat from core client for 30 sec - exiting
06:40:38 (10373): No heartbeat from core client for 30 sec - exiting
06:40:39 (10373): No heartbeat from core client for 30 sec - exiting
06:40:40 (10373): No heartbeat from core client for 30 sec - exiting
06:40:41 (10373): No heartbeat from core client for 30 sec - exiting
06:40:42 (10373): No heartbeat from core client for 30 sec - exiting
06:40:43 (10373): No heartbeat from core client for 30 sec - exiting
06:40:44 (10373): No heartbeat from core client for 30 sec - exiting
06:40:45 (10373): No heartbeat from core client for 30 sec - exiting
06:40:46 (10373): No heartbeat from core client for 30 sec - exiting
06:40:47 (10373): No heartbeat from core client for 30 sec - exiting
06:40:48 (10373): No heartbeat from core client for 30 sec - exiting
06:40:49 (10373): No heartbeat from core client for 30 sec - exiting
06:40:50 (10373): No heartbeat from core client for 30 sec - exiting
06:40:51 (10373): No heartbeat from core client for 30 sec - exiting
06:40:52 (10373): No heartbeat from core client for 30 sec - exiting
06:40:53 (10373): No heartbeat from core client for 30 sec - exiting
06:40:54 (10373): No heartbeat from core client for 30 sec - exiting
06:40:55 (10373): No heartbeat from core client for 30 sec - exiting
06:40:56 (10373): No heartbeat from core client for 30 sec - exiting
06:40:57 (10373): No heartbeat from core client for 30 sec - exiting
06:45:35 (38883): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:45:36 (38883): No heartbeat from core client for 30 sec - exiting
06:45:37 (38883): No heartbeat from core client for 30 sec - exiting
06:45:38 (38883): No heartbeat from core client for 30 sec - exiting
06:45:39 (38883): No heartbeat from core client for 30 sec - exiting
06:45:40 (38883): No heartbeat from core client for 30 sec - exiting
06:45:41 (38883): No heartbeat from core client for 30 sec - exiting
06:45:42 (38883): No heartbeat from core client for 30 sec - exiting
06:45:43 (38883): No heartbeat from core client for 30 sec - exiting
06:45:44 (38883): No heartbeat from core client for 30 sec - exiting
06:45:45 (38883): No heartbeat from core client for 30 sec - exiting
06:45:46 (38883): No heartbeat from core client for 30 sec - exiting
06:45:47 (38883): No heartbeat from core client for 30 sec - exiting
06:45:48 (38883): No heartbeat from core client for 30 sec - exiting
06:45:49 (38883): No heartbeat from core client for 30 sec - exiting
06:45:50 (38883): No heartbeat from core client for 30 sec - exiting
10:36:02 (39084): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:27 (41718): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:40:28 (41718): No heartbeat from core client for 30 sec - exiting
10:40:29 (41718): No heartbeat from core client for 30 sec - exiting
10:40:30 (41718): No heartbeat from core client for 30 sec - exiting
10:40:31 (41718): No heartbeat from core client for 30 sec - exiting
10:40:32 (41718): No heartbeat from core client for 30 sec - exiting
10:40:33 (41718): No heartbeat from core client for 30 sec - exiting
10:40:34 (41718): No heartbeat from core client for 30 sec - exiting
10:40:35 (41718): No heartbeat from core client for 30 sec - exiting
10:40:36 (41718): No heartbeat from core client for 30 sec - exiting
10:40:37 (41718): No heartbeat from core client for 30 sec - exiting
10:40:38 (41718): No heartbeat from core client for 30 sec - exiting
10:40:39 (41718): No heartbeat from core client for 30 sec - exiting
14:03:47 (41930): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:25 (43998): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:13:26 (43998): No heartbeat from core client for 30 sec - exiting
14:13:27 (43998): No heartbeat from core client for 30 sec - exiting
14:13:28 (43998): No heartbeat from core client for 30 sec - exiting
14:13:29 (43998): No heartbeat from core client for 30 sec - exiting
14:51:26 (44372): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
14:51:27 (44372): No heartbeat from core client for 30 sec - exiting
15:05:46 (44883): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:05:47 (44883): No heartbeat from core client for 30 sec - exiting
15:05:48 (44883): No heartbeat from core client for 30 sec - exiting
15:05:49 (44883): No heartbeat from core client for 30 sec - exiting
15:05:50 (44883): No heartbeat from core client for 30 sec - exiting
15:05:51 (44883): No heartbeat from core client for 30 sec - exiting
15:05:52 (44883): No heartbeat from core client for 30 sec - exiting
15:05:53 (44883): No heartbeat from core client for 30 sec - exiting
15:05:54 (44883): No heartbeat from core client for 30 sec - exiting
15:05:55 (44883): No heartbeat from core client for 30 sec - exiting
15:05:56 (44883): No heartbeat from core client for 30 sec - exiting
15:05:57 (44883): No heartbeat from core client for 30 sec - exiting
15:05:58 (44883): No heartbeat from core client for 30 sec - exiting
15:05:59 (44883): No heartbeat from core client for 30 sec - exiting
15:06:00 (44883): No heartbeat from core client for 30 sec - exiting
15:06:01 (44883): No heartbeat from core client for 30 sec - exiting
15:06:02 (44883): No heartbeat from core client for 30 sec - exiting
15:06:03 (44883): No heartbeat from core client for 30 sec - exiting
15:06:04 (44883): No heartbeat from core client for 30 sec - exiting
15:47:25 (45298): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:47:26 (45298): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
16:31:37 (45868): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:31:38 (45868): No heartbeat from core client for 30 sec - exiting
16:31:39 (45868): No heartbeat from core client for 30 sec - exiting
16:31:40 (45868): No heartbeat from core client for 30 sec - exiting
Atmos Hold Restart file rename failed on atmos_restart.hold
17:29:13 (46498): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:29:14 (46498): No heartbeat from core client for 30 sec - exiting
17:29:15 (46498): No heartbeat from core client for 30 sec - exiting
17:29:16 (46498): No heartbeat from core client for 30 sec - exiting
17:29:17 (46498): No heartbeat from core client for 30 sec - exiting
17:29:18 (46498): No heartbeat from core client for 30 sec - exiting
17:29:19 (46498): No heartbeat from core client for 30 sec - exiting
17:29:20 (46498): No heartbeat from core client for 30 sec - exiting
17:29:21 (46498): No heartbeat from core client for 30 sec - exiting
17:29:22 (46498): No heartbeat from core client for 30 sec - exiting
17:29:23 (46498): No heartbeat from core client for 30 sec - exiting
17:29:24 (46498): No heartbeat from core client for 30 sec - exiting
17:38:36 (47406): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:38:37 (47406): No heartbeat from core client for 30 sec - exiting
17:38:38 (47406): No heartbeat from core client for 30 sec - exiting
17:38:39 (47406): No heartbeat from core client for 30 sec - exiting
17:38:40 (47406): No heartbeat from core client for 30 sec - exiting
17:38:41 (47406): No heartbeat from core client for 30 sec - exiting
17:38:42 (47406): No heartbeat from core client for 30 sec - exiting
18:23:44 (47664): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:22 (48388): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:28:23 (48388): No heartbeat from core client for 30 sec - exiting
18:28:24 (48388): No heartbeat from core client for 30 sec - exiting
18:28:25 (48388): No heartbeat from core client for 30 sec - exiting
19:51:15 (48575): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:56:00 (49540): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:00:34 (49732): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
20:01:05 (49732): No heartbeat from core client for 30 sec - exiting
20:01:06 (49732): No heartbeat from core client for 30 sec - exiting
20:01:07 (49732): No heartbeat from core client for 30 sec - exiting
20:01:08 (49732): No heartbeat from core client for 30 sec - exiting
21:12:14 (49922): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:02 (50697): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:17:03 (50697): No heartbeat from core client for 30 sec - exiting
21:17:04 (50697): No heartbeat from core client for 30 sec - exiting
21:17:05 (50697): No heartbeat from core client for 30 sec - exiting
21:17:06 (50697): No heartbeat from core client for 30 sec - exiting
21:17:07 (50697): No heartbeat from core client for 30 sec - exiting
21:17:08 (50697): No heartbeat from core client for 30 sec - exiting
21:17:09 (50697): No heartbeat from core client for 30 sec - exiting
21:17:10 (50697): No heartbeat from core client for 30 sec - exiting
21:17:11 (50697): No heartbeat from core client for 30 sec - exiting
21:17:12 (50697): No heartbeat from core client for 30 sec - exiting
21:17:13 (50697): No heartbeat from core client for 30 sec - exiting
21:17:14 (50697): No heartbeat from core client for 30 sec - exiting
21:17:15 (50697): No heartbeat from core client for 30 sec - exiting
21:17:16 (50697): No heartbeat from core client for 30 sec - exiting
21:17:17 (50697): No heartbeat from core client for 30 sec - exiting
21:17:18 (50697): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7756400]
[0xf7756425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75731df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7576825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf755e4d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7748400]
[0xf7748425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75651df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7568825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75504d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf7761400]
[0xf7761425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf757e1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf7581825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75694d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf773e400]
[0xf773e425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf755b1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf755e825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75464d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf77a0400]
[0xf77a0425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf75bd1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf75c0825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75a84d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called
Stack trace (9 frames):
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f]
[0xf776d400]
[0xf776d425]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xf758a1df]
/lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xf758d825]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395]
/var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xf75754d3]

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50885, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
30 May 2013 16:54:09 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 492,480 1,349,884 2.7410
29 May 2013 19:58:25 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 466,560 1,285,176 2.7546
29 May 2013 01:50:30 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 440,640 1,221,928 2.7731
28 May 2013 07:53:26 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 414,720 1,159,053 2.7948
27 May 2013 11:09:46 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 388,800 1,090,342 2.8044
26 May 2013 15:01:58 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 362,880 1,020,196 2.8114
25 May 2013 19:03:45 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 336,960 949,919 2.8191
24 May 2013 21:58:01 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 311,040 879,339 2.8271
24 May 2013 02:15:42 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 285,120 810,252 2.8418
23 May 2013 06:37:20 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 259,200 741,262 2.8598
22 May 2013 10:03:49 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 233,280 669,984 2.8720
21 May 2013 10:43:47 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 207,360 587,294 2.8322
20 May 2013 10:56:40 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 181,440 504,534 2.7807
19 May 2013 12:18:23 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 155,520 422,268 2.7152
18 May 2013 13:05:12 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 129,600 341,273 2.6333
17 May 2013 16:18:30 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 103,680 269,246 2.5969
16 May 2013 19:52:47 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 77,760 197,144 2.5353
16 May 2013 00:19:12 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 51,840 130,137 2.5104
15 May 2013 05:09:16 1282401 15782964 hadcm3n_4icu_1940_40_008311853_2 25,920 64,232 2.4781


©2024 climateprediction.net