climateprediction.net home page
Task 22295107

Task 22295107

Name oifs_43r3_ps_0826_1984050100_123_953_12170470_1
Workunit 12170470
Created 1 Jan 2023, 18:24:58 UTC
Sent 8 Jan 2023, 10:48:12 UTC
Report deadline 7 Feb 2023, 10:48:12 UTC
Received 12 Jan 2023, 13:28:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1537140
Run time 1 days 13 hours 23 min 24 sec
CPU time 10 hours 55 min 28 sec
Validate state Invalid
Credit 8,467.20
Device peak FLOPS 1.00 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,074.89 MB
Peak swap size 4,978.07 MB
Peak disk usage 3,597.68 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
 the child process
Resuming the child process
  05:59:44 STEP 1879 H=1879:00 +CPU= 16.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:00:05 STEP 1880 H=1880:00 +CPU= 17.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:00:37 STEP 1881 H=1881:00 +CPU= 26.116
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:00 STEP 1882 H=1882:00 +CPU= 17.079
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:21 STEP 1883 H=1883:00 +CPU= 16.588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:01:56 STEP 1884 H=1884:00 +CPU= 28.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001884
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:02:16 STEP 1885 H=1885:00 +CPU= 16.333
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:02:39 STEP 1886 H=1886:00 +CPU= 16.802
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:06 STEP 1887 H=1887:00 +CPU= 22.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:26 STEP 1888 H=1888:00 +CPU= 14.877
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:47 STEP 1889 H=1889:00 +CPU= 15.858
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:04:21 STEP 1890 H=1890:00 +CPU= 26.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:04:42 STEP 1891 H=1891:00 +CPU= 16.390
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:05:12 STEP 1892 H=1892:00 +CPU= 16.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:05:44 STEP 1893 H=1893:00 +CPU= 24.976
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:06:05 STEP 1894 H=1894:00 +CPU= 17.084
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:07:03 STEP 1895 H=1895:00 +CPU= 18.274
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:08:07 STEP 1896 H=1896:00 +CPU= 28.585
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001896
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001884
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001884
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_1984050100_123_953_12170470_1_r427768362_78.zip
Uploading the intermediate file: upload_file_78.zip
  06:08:24 STEP 1897 H=1897:00 +CPU= 16.795
Finished the upload of the intermediate file: upload_file_78.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:08:45 STEP 1898 H=1898:00 +CPU= 16.854
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:17 STEP 1899 H=1899:00 +CPU= 25.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:38 STEP 1900 H=1900:00 +CPU= 17.194
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:58 STEP 1901 H=1901:00 +CPU= 16.356
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:10:33 STEP 1902 H=1902:00 +CPU= 26.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:10:53 STEP 1903 H=1903:00 +CPU= 16.064
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:11:13 STEP 1904 H=1904:00 +CPU= 16.695
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:11:45 STEP 1905 H=1905:00 +CPU= 25.866
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:12:07 STEP 1906 H=1906:00 +CPU= 17.242
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:12:28 STEP 1907 H=1907:00 +CPU= 16.841
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:13:04 STEP 1908 H=1908:00 +CPU= 28.646
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:13:24 STEP 1909 H=1909:00 +CPU= 15.478
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:13:44 STEP 1910 H=1910:00 +CPU= 16.195
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:14:15 STEP 1911 H=1911:00 +CPU= 24.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:14:34 STEP 1912 H=1912:00 +CPU= 13.592
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061440:1673500480:56506.659] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3992MB (heap), 3751MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.074] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.074] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.074] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.074] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.074] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3992 MB (maxheap), 3751 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.159] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-PHYSICS              (1001) 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              EC_PHYS 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               CALLPAR 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                TURBULENCE_LAYER 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 VDFOUTER 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  VDFMAIN 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061441:1673500481:56507.482] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                   VDFEXCU 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061442:1673500482:56508.518] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1723:1723] [20230112:061442:1673500482:56508.518] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001CCB05A  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000FCB934  vdfexcu_                  373  vdfexcu.F90
oifs_43r3_model.e  0000000000FAB617  vdfmain_                  981  vdfmain.F90
oifs_43r3_model.e  0000000000F92699  vdfouter_                 739  vdfouter.F90
oifs_43r3_model.e  0000000000F88512  turbulence_layer_         158  turbulence_layer.F90
oifs_43r3_model.e  0000000000F0B683  callpar_                  869  callpar.F90
oifs_43r3_model.e  0000000000E4940F  ec_phys_                  670  ec_phys.F90
oifs_43r3_model.e  0000000000E3610C  ec_phys_drv_              599  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1984050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0826
(argv4) batchid: 953
(argv5) wuid: 12170470
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/1
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_a239e994dfeeb5e3b51ccb57ac4bc555 to: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0826_1984050100_123_953_12170470.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/1/oifs_43r3_ps_0826_1984050100_123_953_12170470.zip
ic_ancil_file: ic_ancil_12170470
ifsdata_file: ifsdata_12170470
climate_data_file: clim_data_12170470
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_926f6e33cb192b14883e72faaafe4013 to: /var/lib/boinc-client/slots/1/ic_ancil_12170470.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/1/ic_ancil_12170470.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_57dfb7527d2a689a18058ad4484b1a08 to: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12170470.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/1/ifsdata/ifsdata_12170470.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/1/159l_2/clim_data_12170470.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/1/159l_2/clim_data_12170470.zip
Checking for progress XML file: /var/lib/boinc-client/slots/1/progress_file_12170470.xml
Opened progress file ok : /var/lib/boinc-client/slots/1/progress_file_12170470.xml
-- Model is restarting --
Adjusting last_iter, 1912, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/1/progress_file_12170470.xml
last_cpu_time: 37890
upload_file_number: 79
last_iter: 1895
last_upload: 6825600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0826_1984050100_123_953_12170470_1_r427768362
The child process has been launched with process id: 1720
Executing the command: /var/lib/boinc-client/slots/1/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:134814:1673527694:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:49:59 STEP 1896 H=1896:00 +CPU= 24.594
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:52:15 STEP 1897 H=1897:00 +CPU= 17.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:52:39 STEP 1898 H=1898:00 +CPU= 15.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:53:26 STEP 1899 H=1899:00 +CPU= 24.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:53:47 STEP 1900 H=1900:00 +CPU= 15.220
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:54:06 STEP 1901 H=1901:00 +CPU= 15.160
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:54:37 STEP 1902 H=1902:00 +CPU= 25.082
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:54:58 STEP 1903 H=1903:00 +CPU= 16.267
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:55:19 STEP 1904 H=1904:00 +CPU= 16.967
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:55:50 STEP 1905 H=1905:00 +CPU= 25.437
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:11 STEP 1906 H=1906:00 +CPU= 16.838
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:56:32 STEP 1907 H=1907:00 +CPU= 16.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:57:08 STEP 1908 H=1908:00 +CPU= 27.930
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001908
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001908
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:57:28 STEP 1909 H=1909:00 +CPU= 16.217
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:57:47 STEP 1910 H=1910:00 +CPU= 16.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:19 STEP 1911 H=1911:00 +CPU= 25.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:58:40 STEP 1912 H=1912:00 +CPU= 16.905
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:59:00 STEP 1913 H=1913:00 +CPU= 16.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:59:32 STEP 1914 H=1914:00 +CPU= 25.846
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  13:59:55 STEP 1915 H=1915:00 +CPU= 16.270
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:17 STEP 1916 H=1916:00 +CPU= 16.811
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:00:48 STEP 1917 H=1917:00 +CPU= 24.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:01:07 STEP 1918 H=1918:00 +CPU= 14.982
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:01:49 STEP 1919 H=1919:00 +CPU= 16.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:02:59 STEP 1920 H=1920:00 +CPU= 29.531
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001920
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001896
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001908
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001908
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_1984050100_123_953_12170470_1_r427768362_79.zip
Uploading the intermediate file: upload_file_79.zip
  14:03:17 STEP 1921 H=1921:00 +CPU= 16.521
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:04:19 STEP 1922 H=1922:00 +CPU= 17.392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:04:50 STEP 1923 H=1923:00 +CPU= 25.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:05:11 STEP 1924 H=1924:00 +CPU= 16.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:05:31 STEP 1925 H=1925:00 +CPU= 16.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:06:04 STEP 1926 H=1926:00 +CPU= 26.284
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:06:24 STEP 1927 H=1927:00 +CPU= 16.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:06:44 STEP 1928 H=1928:00 +CPU= 16.610
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:07:18 STEP 1929 H=1929:00 +CPU= 25.433
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:07:39 STEP 1930 H=1930:00 +CPU= 16.867
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:07:59 STEP 1931 H=1931:00 +CPU= 16.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:08:34 STEP 1932 H=1932:00 +CPU= 28.406
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001932
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:08:55 STEP 1933 H=1933:00 +CPU= 16.436
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:09:18 STEP 1934 H=1934:00 +CPU= 16.454
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:09:50 STEP 1935 H=1935:00 +CPU= 25.252
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:10:10 STEP 1936 H=1936:00 +CPU= 16.567
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:10:30 STEP 1937 H=1937:00 +CPU= 16.271
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:11:03 STEP 1938 H=1938:00 +CPU= 26.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:11:22 STEP 1939 H=1939:00 +CPU= 15.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:11:41 STEP 1940 H=1940:00 +CPU= 14.516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:13 STEP 1941 H=1941:00 +CPU= 25.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:36 STEP 1942 H=1942:00 +CPU= 17.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:12:58 STEP 1943 H=1943:00 +CPU= 16.755
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:14:03 STEP 1944 H=1944:00 +CPU= 29.310
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001944
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001944
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001920
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMGGhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMSHhq0f+001932
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12170470/ICMUAhq0f+001932
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0826_1984050100_123_953_12170470_1_r427768362_80.zip
Uploading the intermediate file: upload_file_80.zip
  14:14:21 STEP 1945 H=1945:00 +CPU= 16.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:14:40 STEP 1946 H=1946:00 +CPU= 16.995
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:15:43 STEP 1947 H=1947:00 +CPU= 26.883
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:16:04 STEP 1948 H=1948:00 +CPU= 16.722
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:16:25 STEP 1949 H=1949:00 +CPU= 16.597
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:17:03 STEP 1950 H=1950:00 +CPU= 26.464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:17:23 STEP 1951 H=1951:00 +CPU= 16.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:18:08 STEP 1952 H=1952:00 +CPU= 17.580
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:18:42 STEP 1953 H=1953:00 +CPU= 26.316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:07 STEP 1954 H=1954:00 +CPU= 16.362
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:19:27 STEP 1955 H=1955:00 +CPU= 15.718
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:20:20 STEP 1956 H=1956:00 +CPU= 29.873
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMGGhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMSHhq0f+001956
Moving to projects directory: /var/lib/boinc-client/slots/1/ICMUAhq0f+001956
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:21:12 STEP 1957 H=1957:00 +CPU= 19.132
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:21:51 STEP 1958 H=1958:00 +CPU= 17.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:22:23 STEP 1959 H=1959:00 +CPU= 25.793
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:22:43 STEP 1960 H=1960:00 +CPU= 15.779
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:23:00 STEP 1961 H=1961:00 +CPU= 13.335
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:23:33 STEP 1962 H=1962:00 +CPU= 23.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:24:09 STEP 1963 H=1963:00 +CPU= 16.149
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:24:30 STEP 1964 H=1964:00 +CPU= 16.953
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:25:05 STEP 1965 H=1965:00 +CPU= 26.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:25:27 STEP 1966 H=1966:00 +CPU= 17.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  14:25:48 STEP 1967 H=1967:00 +CPU= 16.978
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.202] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3865MB (heap), 3746MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.203] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.219] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.219] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.296] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.296] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 3865 MB (maxheap), 3746 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.360] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.365] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.380] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.380] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            EC_PHYS_DRV 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             >OMP-GPMKTEND             (1063) 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142653:1673530013:2318.390] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              GPMKTEND 
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142654:1673530014:2319.391] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:olaf-LIFEBOOK-U939:1:1:1720:1720] [20230112:142654:1673530014:2319.391] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000E3E156  gpmktend_                 103  gpmktend.F90
oifs_43r3_model.e  0000000000E2EFDA  ec_phys_drv_              279  ec_phys_drv.F90
oifs_43r3_model.e  0000000000BFA3D3  gp_model_                 613  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:15:35 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 10,623,600 59,386 0.0056
09 May 2023 16:14:59 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 10,281,600 57,488 0.0056
09 May 2023 16:14:54 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 9,417,600 52,727 0.0056
09 May 2023 16:14:51 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 8,553,600 47,992 0.0056
09 May 2023 16:14:46 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 7,689,600 43,263 0.0056
09 May 2023 16:14:46 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 7,689,600 43,263 0.0056
09 May 2023 16:13:59 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 6,220,800 35,174 0.0057
09 May 2023 16:16:16 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 6,048,000 33,323 0.0055
09 May 2023 16:13:54 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 5,356,800 30,456 0.0057
09 May 2023 16:13:54 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 5,356,800 30,456 0.0057
09 May 2023 16:16:10 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 5,184,000 28,577 0.0055
09 May 2023 16:13:50 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 4,492,800 25,746 0.0057
09 May 2023 16:13:50 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 4,492,800 25,746 0.0057
09 May 2023 16:16:02 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 4,320,000 23,858 0.0055
09 May 2023 16:13:45 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 3,628,800 21,032 0.0058
09 May 2023 16:13:45 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 3,628,800 21,032 0.0058
09 May 2023 16:15:59 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 3,456,000 19,124 0.0055
09 May 2023 16:13:40 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 2,764,800 16,334 0.0059
09 May 2023 16:15:53 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 2,592,000 14,338 0.0055
09 May 2023 16:12:59 1537140 22295107 oifs_43r3_ps_0826_1984050100_123_953_12170470_1 1,900,800 11,543 0.0061


©2024 climateprediction.net