Task 21873132

Name	hadam4h_a1pi_209811_4_861_011983329_0
Workunit	11983329
Created	8 Jan 2020, 15:54:46 UTC
Sent	10 Jan 2020, 1:55:39 UTC
Report deadline	22 Dec 2020, 7:15:39 UTC
Received	15 Jan 2020, 2:52:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	1492959
Run time	3 days 21 hours 18 min 15 sec
CPU time	3 days 20 hours 44 min 14 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office HadAM4 at N216 resolution v8.52 i686-pc-linux-gnu
Peak working set size	1,364.63 MB
Peak swap size	1,385.75 MB
Peak disk usage	12.91 MB
Stderr	<core_client_version>7.9.3</core_client_version> <![CDATA[ <message> process exited with code 12 (0xc, -244)</message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xnnuj.pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA Signal 15 received: Software termination signal from kill Signal 15 received: Abnormal termination triggered by abort call Signal 15 received, exiting... SIGSEGV: segmentation violation Stack trace (17 frames): ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu(boinc_catch_signal+0x67)[0x80d4cf7] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f2a090] /lib32/libc.so.6(getenv+0x99)[0xf7ad05f9] /lib32/libc.so.6(+0xae498)[0xf7b4f498] /lib32/libc.so.6(+0xae865)[0xf7b4f865] /lib32/libc.so.6(localtime_r+0x12)[0xf7b4ddc2] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d01b2] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d0900] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d09f1] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf7f2a090] linux-gate.so.1(__kernel_vsyscall+0x9)[0xf7f2a079] /lib32/libc.so.6(nanosleep+0x4b)[0xf7b5ef6b] /lib32/libc.so.6(usleep+0x41)[0xf7b911a1] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80e78a5] ../../projects/climateprediction.net/hadam4_8.52_i686-pc-linux-gnu[0x80d2114] /lib32/libpthread.so.0(+0x63a6)[0xf7ef43a6] /lib32/libc.so.6(clone+0x66)[0xf7b98396] Exiting... OPEN: File Creation Failed: No such file or directory OPEN: Unable to Open File dataout/a1piga.pat8nov for Read/Write Model crashed: STWORK : Error opening output PP file on unit 60 tmp/xnnuj.pipe_dummy cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/7692/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2891, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/7700/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2891, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/7708/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2891, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.ihist after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/jobs/xnnuj.namelists after 11 attempts cpdnmonitor: cannot open input file /var/lib/boinc-client/projects/climateprediction.net/hadam4h_a1pi_209811_4_861_011983329/dataout/atmos_restart.day after 11 attempts forrtl: Bad file descriptor forrtl: severe (30): open failure, unit 6, file /proc/7718/fd/ Image PC Routine Line Source hadam4_um_8.52_i6 083F6605 Unknown Unknown Unknown hadam4_um_8.52_i6 0843E0A0 Unknown Unknown Unknown hadam4_um_8.52_i6 081DF6C9 Unknown Unknown Unknown hadam4_um_8.52_i6 0836E63F Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2891, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( </stderr_txt> ]]>