climateprediction.net home page
Task 15835215

Task 15835215

Name hadcm3n_n2zy_1880_40_008376961_1
Workunit 8527820
Created 8 Jun 2013, 9:26:17 UTC
Sent 8 Jun 2013, 9:40:28 UTC
Report deadline 7 Sep 2013, 17:07:39 UTC
Received 29 Jun 2013, 11:00:57 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1283984
Run time 16 days 9 hours 56 min 41 sec
CPU time 12 days 12 hours 50 min 10 sec
Validate state Invalid
Credit 4,976.64
Device peak FLOPS 0.95 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr 
<core_client_version>7.0.65</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
23:34:17 (1642): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
21:50:32 (1507): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
21:51:02 (1507): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
23:22:31 (12954): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
23:22:43 (12954): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
22:35:49 (1627): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:35:59 (1627): No heartbeat from core client for 30 sec - exiting
22:36:09 (1627): No heartbeat from core client for 30 sec - exiting
22:38:19 (24426): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:38:30 (24426): No heartbeat from core client for 30 sec - exiting
22:41:07 (24451): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
22:41:16 (24451): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
OPEN:  File Creation Failed: Too many open files in system
OPEN:  Unable to Open File dataout/n2zyko.da89cb0 for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_se_6.07_i686-pc-linux-gnu.so after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.ihist after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.namelists after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/ocean_restart.day after 11 attempts
SIGSEGV: segmentation violation
Stack trace (13 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7789400]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc527]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc8df]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc9a9]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bcbdf]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b4126]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b0ff7]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80507a4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051127]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/libc.so.6(__libc_start_main+0xf5)[0x4a237865]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
OPEN:  File Creation Failed: Too many open files in system
OPEN:  Unable to Open File dataout/n2zyka.da901q0 for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_se_6.07_i686-pc-linux-gnu.so after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu after 11 attempts
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish
OPEN:  File Creation Failed: Too many open files in system
OPEN:  Unable to Open File dataout/n2zyka.da91640 for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_se_6.07_i686-pc-linux-gnu.so after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.ihist after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.namelists after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/ocean_restart.day after 11 attempts
SIGSEGV: segmentation violation
Stack trace (13 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7764400]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc527]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc8df]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc9a9]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bcbdf]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b4126]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b0ff7]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80507a4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051127]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/libc.so.6(__libc_start_main+0xf5)[0x4a237865]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...
CPDN Monitor - Quit request from BOINC...
OPEN:  File Creation Failed: Too many open files in system
OPEN:  Unable to Open File dataout/n2zyka.da91690 for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  2048    
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_se_6.07_i686-pc-linux-gnu.so after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.ihist after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/jobs/xabnk.namelists after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/atmos_restart.day after 11 attempts
cpdnmonitor: cannot open input file /var/lib/boinc/projects/climateprediction.net/hadcm3n_n2zy_1880_40_008376961/dataout/ocean_restart.day after 11 attempts
SIGSEGV: segmentation violation
Stack trace (13 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7722400]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc527]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc8df]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bc9a9]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80bcbdf]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b4126]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80b0ff7]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x80507a4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8051127]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib/libc.so.6(__libc_start_main+0xf5)[0x4a237865]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...
Signal 15 received, exiting...
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
Signal 15 received, exiting...
Called boinc_finish
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Signal 15 received, exiting...
Called boinc_finish

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFOUT: Write Failed: No space left on device
BUFFOUT: C I/O Error - Return code = 1

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                     Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2204, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
28 Jun 2013 14:56:06 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 414,720 1,150,412 2.7739
27 Jun 2013 16:12:43 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 388,800 1,074,014 2.7624
26 Jun 2013 16:13:30 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 362,880 992,324 2.7346
25 Jun 2013 16:05:50 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 336,960 909,087 2.6979
24 Jun 2013 12:31:39 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 311,040 889,997 2.8614
23 Jun 2013 07:36:13 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 285,120 798,061 2.7990
19 Jun 2013 05:50:00 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 259,200 709,587 2.7376
16 Jun 2013 20:07:47 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 233,280 618,159 2.6499
15 Jun 2013 14:37:06 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 207,360 548,374 2.6446
14 Jun 2013 14:21:08 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 181,440 479,350 2.6419
13 Jun 2013 17:56:04 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 155,520 410,085 2.6369
12 Jun 2013 19:05:25 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 129,600 341,688 2.6365
11 Jun 2013 21:09:18 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 103,680 272,879 2.6319
10 Jun 2013 23:44:09 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 77,760 204,505 2.6300
10 Jun 2013 03:10:27 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 51,840 136,534 2.6338
09 Jun 2013 06:43:53 1283984 15835215 hadcm3n_n2zy_1880_40_008376961_1 25,920 68,272 2.6340


©2024 cpdn.org