climateprediction.net home page
Task 22281711

Task 22281711

Name oifs_43r3_ps_0124_2010050100_123_979_12195768_0
Workunit 12195768
Created 23 Dec 2022, 22:51:21 UTC
Sent 23 Dec 2022, 22:59:42 UTC
Report deadline 22 Jan 2023, 22:59:42 UTC
Received 22 Jan 2023, 7:12:14 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1507869
Run time 20 hours 42 min 37 sec
CPU time 15 hours 29 min 20 sec
Validate state Invalid
Credit 3,870.72
Device peak FLOPS 3.97 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,404.13 MB
Peak swap size 4,978.07 MB
Peak disk usage 4,442.16 MB
Stderr 
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
.006] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.007] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.007] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.007] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.008] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:51718:51718] [20230121:174846:1674344926:0.008] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:51:20 STEP  984 H= 984:00 +CPU= 60.690
  17:52:10 STEP  985 H= 985:00 +CPU= 34.960
  17:53:04 STEP  986 H= 986:00 +CPU= 35.523
  17:54:25 STEP  987 H= 987:00 +CPU= 53.322
  17:55:23 STEP  988 H= 988:00 +CPU= 35.832
  17:56:11 STEP  989 H= 989:00 +CPU= 34.909
  17:57:25 STEP  990 H= 990:00 +CPU= 53.874
  17:58:14 STEP  991 H= 991:00 +CPU= 34.699
  17:59:05 STEP  992 H= 992:00 +CPU= 35.479
  18:00:34 STEP  993 H= 993:00 +CPU= 53.237
  18:01:43 STEP  994 H= 994:00 +CPU= 35.833
  18:02:45 STEP  995 H= 995:00 +CPU= 34.865
  18:04:23 STEP  996 H= 996:00 +CPU= 59.841
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+000996
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+000996
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+000996
  18:05:10 STEP  997 H= 997:00 +CPU= 34.787
  18:05:59 STEP  998 H= 998:00 +CPU= 35.805
  18:07:15 STEP  999 H= 999:00 +CPU= 53.356
  18:08:01 STEP 1000 H=1000:00 +CPU= 35.558
  18:08:50 STEP 1001 H=1001:00 +CPU= 34.875
  18:10:04 STEP 1002 H=1002:00 +CPU= 54.586
  18:10:51 STEP 1003 H=1003:00 +CPU= 34.751
  18:11:41 STEP 1004 H=1004:00 +CPU= 35.364
  18:13:03 STEP 1005 H=1005:00 +CPU= 52.987
  18:13:54 STEP 1006 H=1006:00 +CPU= 35.061
  18:14:44 STEP 1007 H=1007:00 +CPU= 35.615
  18:16:34 STEP 1008 H=1008:00 +CPU= 60.290
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001008
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001008
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001008
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+000984
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+000984
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+000984
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+000996
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+000996
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+000996
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_41.zip
Uploading the intermediate file: upload_file_41.zip
  18:17:28 STEP 1009 H=1009:00 +CPU= 35.003
  18:18:21 STEP 1010 H=1010:00 +CPU= 35.244
  18:19:35 STEP 1011 H=1011:00 +CPU= 52.223
  18:20:25 STEP 1012 H=1012:00 +CPU= 35.023
  18:21:15 STEP 1013 H=1013:00 +CPU= 33.864
  18:22:30 STEP 1014 H=1014:00 +CPU= 53.159
  18:23:18 STEP 1015 H=1015:00 +CPU= 34.052
  18:24:13 STEP 1016 H=1016:00 +CPU= 35.223
  18:25:33 STEP 1017 H=1017:00 +CPU= 52.031
  18:26:26 STEP 1018 H=1018:00 +CPU= 35.075
  18:27:16 STEP 1019 H=1019:00 +CPU= 34.256
  18:28:36 STEP 1020 H=1020:00 +CPU= 58.422
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001020
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001020
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001020
  18:29:36 STEP 1021 H=1021:00 +CPU= 34.892
  18:30:37 STEP 1022 H=1022:00 +CPU= 35.822
  18:32:07 STEP 1023 H=1023:00 +CPU= 53.835
  18:33:06 STEP 1024 H=1024:00 +CPU= 35.974
  18:34:06 STEP 1025 H=1025:00 +CPU= 35.367
  18:35:36 STEP 1026 H=1026:00 +CPU= 54.896
  18:36:34 STEP 1027 H=1027:00 +CPU= 35.100
  18:37:36 STEP 1028 H=1028:00 +CPU= 35.937
  18:38:45 STEP 1029 H=1029:00 +CPU= 52.766
  18:39:32 STEP 1030 H=1030:00 +CPU= 35.134
  18:40:19 STEP 1031 H=1031:00 +CPU= 35.430
  18:42:01 STEP 1032 H=1032:00 +CPU= 61.581
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001032
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001032
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001032
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001008
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001008
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001008
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001020
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001020
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001020
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_42.zip
Uploading the intermediate file: upload_file_42.zip
  18:42:58 STEP 1033 H=1033:00 +CPU= 34.551
  18:44:00 STEP 1034 H=1034:00 +CPU= 36.379
  18:45:30 STEP 1035 H=1035:00 +CPU= 54.663
  18:46:28 STEP 1036 H=1036:00 +CPU= 35.401
  18:47:26 STEP 1037 H=1037:00 +CPU= 35.175
  18:48:59 STEP 1038 H=1038:00 +CPU= 56.262
  18:49:56 STEP 1039 H=1039:00 +CPU= 34.404
  18:50:55 STEP 1040 H=1040:00 +CPU= 35.755
  18:52:24 STEP 1041 H=1041:00 +CPU= 54.560
  18:53:23 STEP 1042 H=1042:00 +CPU= 35.670
  18:54:24 STEP 1043 H=1043:00 +CPU= 35.628
  18:56:05 STEP 1044 H=1044:00 +CPU= 60.886
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001044
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001044
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001044
  18:57:02 STEP 1045 H=1045:00 +CPU= 34.699
  18:58:01 STEP 1046 H=1046:00 +CPU= 35.717
  18:59:20 STEP 1047 H=1047:00 +CPU= 54.406
  19:00:08 STEP 1048 H=1048:00 +CPU= 33.924
  19:00:59 STEP 1049 H=1049:00 +CPU= 34.416
  19:02:18 STEP 1050 H=1050:00 +CPU= 54.661
  19:03:03 STEP 1051 H=1051:00 +CPU= 32.661
  19:03:54 STEP 1052 H=1052:00 +CPU= 35.234
  19:05:14 STEP 1053 H=1053:00 +CPU= 52.631
  19:06:06 STEP 1054 H=1054:00 +CPU= 33.630
  19:07:01 STEP 1055 H=1055:00 +CPU= 35.119
  19:08:35 STEP 1056 H=1056:00 +CPU= 60.617
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001056
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001056
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001056
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001032
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001032
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001032
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001044
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001044
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001044
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_43.zip
Uploading the intermediate file: upload_file_43.zip
  19:09:23 STEP 1057 H=1057:00 +CPU= 33.169
  19:10:17 STEP 1058 H=1058:00 +CPU= 35.225
  19:11:35 STEP 1059 H=1059:00 +CPU= 53.014
  19:12:28 STEP 1060 H=1060:00 +CPU= 34.401
  19:13:19 STEP 1061 H=1061:00 +CPU= 34.112
  19:14:43 STEP 1062 H=1062:00 +CPU= 53.918
  19:15:36 STEP 1063 H=1063:00 +CPU= 33.749
  19:16:30 STEP 1064 H=1064:00 +CPU= 35.073
  19:17:52 STEP 1065 H=1065:00 +CPU= 52.813
  19:18:44 STEP 1066 H=1066:00 +CPU= 34.381
  19:19:35 STEP 1067 H=1067:00 +CPU= 33.797
  19:21:03 STEP 1068 H=1068:00 +CPU= 58.701
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001068
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001068
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001068
  19:21:49 STEP 1069 H=1069:00 +CPU= 33.432
  19:22:44 STEP 1070 H=1070:00 +CPU= 34.932
  19:24:05 STEP 1071 H=1071:00 +CPU= 53.308
  19:24:59 STEP 1072 H=1072:00 +CPU= 34.399
  19:25:47 STEP 1073 H=1073:00 +CPU= 33.688
  19:27:00 STEP 1074 H=1074:00 +CPU= 53.801
  19:27:45 STEP 1075 H=1075:00 +CPU= 33.631
  19:28:33 STEP 1076 H=1076:00 +CPU= 35.293
  19:29:49 STEP 1077 H=1077:00 +CPU= 53.893
  19:30:37 STEP 1078 H=1078:00 +CPU= 34.569
  19:31:29 STEP 1079 H=1079:00 +CPU= 35.062
  19:33:10 STEP 1080 H=1080:00 +CPU= 62.586
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001080
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001080
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001080
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001056
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001056
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001056
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001068
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001068
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001068
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_44.zip
Uploading the intermediate file: upload_file_44.zip
  19:33:56 STEP 1081 H=1081:00 +CPU= 33.168
  19:34:43 STEP 1082 H=1082:00 +CPU= 34.848
  19:36:05 STEP 1083 H=1083:00 +CPU= 53.706
  19:36:58 STEP 1084 H=1084:00 +CPU= 34.762
  19:37:52 STEP 1085 H=1085:00 +CPU= 34.032
  19:39:23 STEP 1086 H=1086:00 +CPU= 55.038
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0124
(argv4) batchid: 979
(argv5) wuid: 12195768
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/Cache/Linux/boinc-client/slots/12
Project directory is: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768
..mkdir for temp folder for results failed
Copying: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7396b335c278c8c9540a86cf57f851ca to: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_0124_2010050100_123_979_12195768.zip
Unzipping the namelist zip file: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_0124_2010050100_123_979_12195768.zip
ic_ancil_file: ic_ancil_12195768
ifsdata_file: ifsdata_12195768
climate_data_file: clim_data_12195768
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_70c068e9b28cb2c4e30c20c238cca379 to: /mnt/Cache/Linux/boinc-client/slots/12/ic_ancil_12195768.zip
Unzipping the IC ancils zip file: /mnt/Cache/Linux/boinc-client/slots/12/ic_ancil_12195768.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f69ab474dd020aa7c8fc3c8e760a8a1f to: /mnt/Cache/Linux/boinc-client/slots/12/ifsdata/ifsdata_12195768.zip
Unzipping the ifsdata_zip file: /mnt/Cache/Linux/boinc-client/slots/12/ifsdata/ifsdata_12195768.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/Cache/Linux/boinc-client/slots/12/159l_2/clim_data_12195768.zip
Unzipping the climate data zip file: /mnt/Cache/Linux/boinc-client/slots/12/159l_2/clim_data_12195768.zip
Checking for progress XML file: /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
Opened progress file ok : /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
-- Model is restarting --
Adjusting last_iter, 1087, to previous model restart step.
Creating progress file: /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
last_cpu_time: 45647
upload_file_number: 45
last_iter: 1079
last_upload: 3888000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781
The child process has been launched with process id: 61729
Executing the command: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.002] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.013] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.014] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.015] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.015] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.015] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.015] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:61729:61729] [20230121:195811:1674352691:0.015] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:01:18 STEP 1080 H=1080:00 +CPU= 59.515
  20:02:28 STEP 1081 H=1081:00 +CPU= 33.837
  20:03:25 STEP 1082 H=1082:00 +CPU= 35.017
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  20:05:00 STEP 1083 H=1083:00 +CPU= 52.757
  20:05:55 STEP 1084 H=1084:00 +CPU= 35.502
  20:06:46 STEP 1085 H=1085:00 +CPU= 34.657
  20:08:05 STEP 1086 H=1086:00 +CPU= 53.640
  20:09:01 STEP 1087 H=1087:00 +CPU= 34.314
  20:10:00 STEP 1088 H=1088:00 +CPU= 35.398
  20:11:30 STEP 1089 H=1089:00 +CPU= 52.826
  20:12:21 STEP 1090 H=1090:00 +CPU= 35.000
  20:13:09 STEP 1091 H=1091:00 +CPU= 33.660
  20:14:31 STEP 1092 H=1092:00 +CPU= 58.540
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001092
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001092
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001092
  20:15:20 STEP 1093 H=1093:00 +CPU= 34.401
  20:16:14 STEP 1094 H=1094:00 +CPU= 35.484
  20:17:32 STEP 1095 H=1095:00 +CPU= 52.938
  20:18:27 STEP 1096 H=1096:00 +CPU= 34.803
  20:19:20 STEP 1097 H=1097:00 +CPU= 33.966
  20:20:43 STEP 1098 H=1098:00 +CPU= 53.185
  20:21:37 STEP 1099 H=1099:00 +CPU= 34.415
  20:22:33 STEP 1100 H=1100:00 +CPU= 35.214
  20:24:04 STEP 1101 H=1101:00 +CPU= 53.512
  20:25:02 STEP 1102 H=1102:00 +CPU= 35.158
  20:26:00 STEP 1103 H=1103:00 +CPU= 34.903
  20:28:17 STEP 1104 H=1104:00 +CPU= 62.196
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001104
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001104
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001104
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001080
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001080
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001080
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001092
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001092
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001092
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_45.zip
Uploading the intermediate file: upload_file_45.zip
  20:29:14 STEP 1105 H=1105:00 +CPU= 34.122
  20:30:10 STEP 1106 H=1106:00 +CPU= 35.453
  20:31:35 STEP 1107 H=1107:00 +CPU= 53.106
  20:32:33 STEP 1108 H=1108:00 +CPU= 35.199
  20:33:29 STEP 1109 H=1109:00 +CPU= 34.768
  20:34:55 STEP 1110 H=1110:00 +CPU= 53.925
  20:35:48 STEP 1111 H=1111:00 +CPU= 34.400
  20:36:41 STEP 1112 H=1112:00 +CPU= 35.766
  20:38:02 STEP 1113 H=1113:00 +CPU= 52.844
  20:39:01 STEP 1114 H=1114:00 +CPU= 34.925
  20:39:57 STEP 1115 H=1115:00 +CPU= 34.101
  20:41:29 STEP 1116 H=1116:00 +CPU= 58.950
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001116
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001116
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001116
  20:42:22 STEP 1117 H=1117:00 +CPU= 35.145
  20:43:14 STEP 1118 H=1118:00 +CPU= 35.156
  20:44:38 STEP 1119 H=1119:00 +CPU= 52.915
  20:45:31 STEP 1120 H=1120:00 +CPU= 35.813
  20:46:25 STEP 1121 H=1121:00 +CPU= 34.931
  20:47:46 STEP 1122 H=1122:00 +CPU= 53.616
  20:48:38 STEP 1123 H=1123:00 +CPU= 34.872
  20:49:33 STEP 1124 H=1124:00 +CPU= 35.046
  20:50:59 STEP 1125 H=1125:00 +CPU= 52.462
  20:51:54 STEP 1126 H=1126:00 +CPU= 35.316
  20:52:52 STEP 1127 H=1127:00 +CPU= 34.716
  20:54:51 STEP 1128 H=1128:00 +CPU= 62.431
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001128
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001128
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001128
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001104
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001104
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001104
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001116
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001116
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001116
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_46.zip
Uploading the intermediate file: upload_file_46.zip
  20:55:45 STEP 1129 H=1129:00 +CPU= 34.375
  20:56:34 STEP 1130 H=1130:00 +CPU= 35.717
  20:57:53 STEP 1131 H=1131:00 +CPU= 53.791
  20:58:48 STEP 1132 H=1132:00 +CPU= 35.050
  20:59:39 STEP 1133 H=1133:00 +CPU= 33.533
  21:01:04 STEP 1134 H=1134:00 +CPU= 52.608
  21:02:00 STEP 1135 H=1135:00 +CPU= 34.057
  21:02:54 STEP 1136 H=1136:00 +CPU= 35.600
  21:04:17 STEP 1137 H=1137:00 +CPU= 54.527
  21:05:11 STEP 1138 H=1138:00 +CPU= 35.625
  21:06:04 STEP 1139 H=1139:00 +CPU= 35.132
  21:07:37 STEP 1140 H=1140:00 +CPU= 59.753
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001140
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001140
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001140
  21:08:29 STEP 1141 H=1141:00 +CPU= 34.878
  21:09:23 STEP 1142 H=1142:00 +CPU= 35.728
  21:10:44 STEP 1143 H=1143:00 +CPU= 53.937
  21:11:40 STEP 1144 H=1144:00 +CPU= 35.722
  21:12:34 STEP 1145 H=1145:00 +CPU= 35.051
  21:13:57 STEP 1146 H=1146:00 +CPU= 55.066
  21:14:51 STEP 1147 H=1147:00 +CPU= 35.071
  21:15:45 STEP 1148 H=1148:00 +CPU= 35.962
  21:17:06 STEP 1149 H=1149:00 +CPU= 54.213
  21:18:00 STEP 1150 H=1150:00 +CPU= 35.293
  21:18:54 STEP 1151 H=1151:00 +CPU= 35.220
  21:20:57 STEP 1152 H=1152:00 +CPU= 63.933
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001152
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001152
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001152
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001128
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001128
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001128
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001140
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001140
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001140
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_47.zip
Uploading the intermediate file: upload_file_47.zip
  21:21:50 STEP 1153 H=1153:00 +CPU= 34.660
  21:22:42 STEP 1154 H=1154:00 +CPU= 35.230
  21:23:55 STEP 1155 H=1155:00 +CPU= 54.008
  21:24:42 STEP 1156 H=1156:00 +CPU= 35.144
  21:25:28 STEP 1157 H=1157:00 +CPU= 34.513
  21:26:42 STEP 1158 H=1158:00 +CPU= 55.458
  21:27:29 STEP 1159 H=1159:00 +CPU= 34.902
  21:28:14 STEP 1160 H=1160:00 +CPU= 35.112
  21:29:24 STEP 1161 H=1161:00 +CPU= 52.855
  21:30:10 STEP 1162 H=1162:00 +CPU= 35.460
  21:30:56 STEP 1163 H=1163:00 +CPU= 34.829
  21:32:14 STEP 1164 H=1164:00 +CPU= 59.174
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001164
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001164
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001164
  21:32:59 STEP 1165 H=1165:00 +CPU= 34.516
  21:33:47 STEP 1166 H=1166:00 +CPU= 35.926
  21:34:56 STEP 1167 H=1167:00 +CPU= 52.988
  21:35:43 STEP 1168 H=1168:00 +CPU= 34.986
  21:36:29 STEP 1169 H=1169:00 +CPU= 35.163
  21:37:54 STEP 1170 H=1170:00 +CPU= 55.077
  21:38:46 STEP 1171 H=1171:00 +CPU= 34.999
  21:39:39 STEP 1172 H=1172:00 +CPU= 35.590
  21:41:00 STEP 1173 H=1173:00 +CPU= 53.846
  21:41:56 STEP 1174 H=1174:00 +CPU= 35.329
  21:42:50 STEP 1175 H=1175:00 +CPU= 34.817
  21:44:54 STEP 1176 H=1176:00 +CPU= 64.126
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001176
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001176
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001176
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001152
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001152
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001152
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001164
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001164
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001164
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_48.zip
Uploading the intermediate file: upload_file_48.zip
  21:45:47 STEP 1177 H=1177:00 +CPU= 34.609
  21:46:39 STEP 1178 H=1178:00 +CPU= 35.381
  21:47:58 STEP 1179 H=1179:00 +CPU= 53.702
  21:48:51 STEP 1180 H=1180:00 +CPU= 35.120
  21:49:43 STEP 1181 H=1181:00 +CPU= 34.453
  21:51:04 STEP 1182 H=1182:00 +CPU= 54.916
  21:51:56 STEP 1183 H=1183:00 +CPU= 35.092
  21:52:49 STEP 1184 H=1184:00 +CPU= 35.385
  21:54:11 STEP 1185 H=1185:00 +CPU= 54.045
  21:55:03 STEP 1186 H=1186:00 +CPU= 35.203
  21:55:53 STEP 1187 H=1187:00 +CPU= 34.189
  21:57:24 STEP 1188 H=1188:00 +CPU= 59.983
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001188
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001188
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001188
  21:58:17 STEP 1189 H=1189:00 +CPU= 34.596
  21:59:12 STEP 1190 H=1190:00 +CPU= 35.483
  22:00:33 STEP 1191 H=1191:00 +CPU= 53.430
  22:01:24 STEP 1192 H=1192:00 +CPU= 34.917
  22:02:14 STEP 1193 H=1193:00 +CPU= 34.272
  22:03:37 STEP 1194 H=1194:00 +CPU= 54.997
  22:04:30 STEP 1195 H=1195:00 +CPU= 34.695
  22:05:23 STEP 1196 H=1196:00 +CPU= 35.186
  22:06:44 STEP 1197 H=1197:00 +CPU= 53.851
  22:07:36 STEP 1198 H=1198:00 +CPU= 34.635
  22:08:30 STEP 1199 H=1199:00 +CPU= 34.861
  22:10:33 STEP 1200 H=1200:00 +CPU= 63.335
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001200
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001200
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001200
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001176
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001176
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001176
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001188
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001188
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001188
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_49.zip
Uploading the intermediate file: upload_file_49.zip
  22:11:25 STEP 1201 H=1201:00 +CPU= 34.213
  22:12:20 STEP 1202 H=1202:00 +CPU= 35.373
  22:13:39 STEP 1203 H=1203:00 +CPU= 53.171
  22:14:33 STEP 1204 H=1204:00 +CPU= 35.821
  22:15:26 STEP 1205 H=1205:00 +CPU= 35.375
  22:16:47 STEP 1206 H=1206:00 +CPU= 54.074
  22:17:40 STEP 1207 H=1207:00 +CPU= 34.890
  22:18:33 STEP 1208 H=1208:00 +CPU= 35.594
  22:19:52 STEP 1209 H=1209:00 +CPU= 53.568
  22:20:46 STEP 1210 H=1210:00 +CPU= 35.787
  22:21:36 STEP 1211 H=1211:00 +CPU= 34.545
  22:23:09 STEP 1212 H=1212:00 +CPU= 60.954
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001212
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001212
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001212
  22:24:01 STEP 1213 H=1213:00 +CPU= 34.909
  22:24:57 STEP 1214 H=1214:00 +CPU= 35.622
  22:26:17 STEP 1215 H=1215:00 +CPU= 54.108
  22:27:11 STEP 1216 H=1216:00 +CPU= 35.425
  22:28:03 STEP 1217 H=1217:00 +CPU= 34.465
  22:29:26 STEP 1218 H=1218:00 +CPU= 54.961
  22:30:18 STEP 1219 H=1219:00 +CPU= 34.558
  22:31:12 STEP 1220 H=1220:00 +CPU= 35.576
  22:32:33 STEP 1221 H=1221:00 +CPU= 53.773
  22:33:27 STEP 1222 H=1222:00 +CPU= 35.463
  22:34:19 STEP 1223 H=1223:00 +CPU= 35.219
  22:36:05 STEP 1224 H=1224:00 +CPU= 63.412
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001224
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001224
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001224
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001200
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001200
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001200
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001212
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001212
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001212
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_50.zip
Uploading the intermediate file: upload_file_50.zip
  22:36:52 STEP 1225 H=1225:00 +CPU= 34.470
  22:37:40 STEP 1226 H=1226:00 +CPU= 35.999
  22:38:48 STEP 1227 H=1227:00 +CPU= 52.814
  22:39:38 STEP 1228 H=1228:00 +CPU= 34.848
  22:40:29 STEP 1229 H=1229:00 +CPU= 34.767
  22:41:55 STEP 1230 H=1230:00 +CPU= 55.067
  22:42:48 STEP 1231 H=1231:00 +CPU= 34.685
  22:43:43 STEP 1232 H=1232:00 +CPU= 36.124
  22:45:04 STEP 1233 H=1233:00 +CPU= 54.050
  22:45:58 STEP 1234 H=1234:00 +CPU= 35.290
  22:46:50 STEP 1235 H=1235:00 +CPU= 34.918
  22:48:21 STEP 1236 H=1236:00 +CPU= 60.253
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001236
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001236
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001236
  22:49:13 STEP 1237 H=1237:00 +CPU= 34.502
  22:50:08 STEP 1238 H=1238:00 +CPU= 35.677
  22:51:26 STEP 1239 H=1239:00 +CPU= 53.091
  22:52:20 STEP 1240 H=1240:00 +CPU= 35.832
  22:53:13 STEP 1241 H=1241:00 +CPU= 34.859
  22:54:37 STEP 1242 H=1242:00 +CPU= 55.046
  22:55:29 STEP 1243 H=1243:00 +CPU= 35.195
  22:56:23 STEP 1244 H=1244:00 +CPU= 35.456
  22:57:45 STEP 1245 H=1245:00 +CPU= 53.330
  22:58:40 STEP 1246 H=1246:00 +CPU= 35.532
  22:59:35 STEP 1247 H=1247:00 +CPU= 34.837
  23:01:35 STEP 1248 H=1248:00 +CPU= 62.780
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001248
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001248
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001248
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001224
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001224
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001224
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001236
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001236
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001236
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_51.zip
Uploading the intermediate file: upload_file_51.zip
  23:02:28 STEP 1249 H=1249:00 +CPU= 34.669
  23:03:23 STEP 1250 H=1250:00 +CPU= 35.777
  23:04:43 STEP 1251 H=1251:00 +CPU= 53.593
  23:05:37 STEP 1252 H=1252:00 +CPU= 35.288
  23:06:31 STEP 1253 H=1253:00 +CPU= 34.728
  23:07:53 STEP 1254 H=1254:00 +CPU= 53.981
  23:08:44 STEP 1255 H=1255:00 +CPU= 34.507
  23:09:40 STEP 1256 H=1256:00 +CPU= 36.098
  23:10:59 STEP 1257 H=1257:00 +CPU= 53.650
  23:11:57 STEP 1258 H=1258:00 +CPU= 36.107
  23:12:50 STEP 1259 H=1259:00 +CPU= 34.476
  23:14:22 STEP 1260 H=1260:00 +CPU= 59.540
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001260
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001260
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001260
  23:15:18 STEP 1261 H=1261:00 +CPU= 34.915
  23:16:12 STEP 1262 H=1262:00 +CPU= 36.044
  23:17:31 STEP 1263 H=1263:00 +CPU= 53.360
  23:18:24 STEP 1264 H=1264:00 +CPU= 35.724
  23:19:17 STEP 1265 H=1265:00 +CPU= 34.880
  23:20:38 STEP 1266 H=1266:00 +CPU= 54.302
  23:21:31 STEP 1267 H=1267:00 +CPU= 35.433
  23:22:25 STEP 1268 H=1268:00 +CPU= 35.279
  23:23:44 STEP 1269 H=1269:00 +CPU= 53.028
  23:24:39 STEP 1270 H=1270:00 +CPU= 35.201
  23:25:35 STEP 1271 H=1271:00 +CPU= 35.780
  23:27:37 STEP 1272 H=1272:00 +CPU= 63.746
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001272
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001272
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001272
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001248
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001248
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001248
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001260
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001260
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001260
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_52.zip
Uploading the intermediate file: upload_file_52.zip
  23:28:31 STEP 1273 H=1273:00 +CPU= 35.026
  23:29:25 STEP 1274 H=1274:00 +CPU= 35.864
  23:30:46 STEP 1275 H=1275:00 +CPU= 53.634
  23:31:40 STEP 1276 H=1276:00 +CPU= 36.065
  23:32:33 STEP 1277 H=1277:00 +CPU= 34.806
  23:33:56 STEP 1278 H=1278:00 +CPU= 55.060
  23:34:47 STEP 1279 H=1279:00 +CPU= 34.265
  23:35:42 STEP 1280 H=1280:00 +CPU= 35.745
  23:36:56 STEP 1281 H=1281:00 +CPU= 53.154
  23:37:48 STEP 1282 H=1282:00 +CPU= 35.529
  23:38:36 STEP 1283 H=1283:00 +CPU= 35.039
  23:39:56 STEP 1284 H=1284:00 +CPU= 59.349
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001284
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001284
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001284
  23:40:43 STEP 1285 H=1285:00 +CPU= 34.717
  23:41:31 STEP 1286 H=1286:00 +CPU= 35.541
  23:42:45 STEP 1287 H=1287:00 +CPU= 53.532
  23:43:34 STEP 1288 H=1288:00 +CPU= 35.999
  23:44:24 STEP 1289 H=1289:00 +CPU= 34.845
  23:45:44 STEP 1290 H=1290:00 +CPU= 54.935
  23:46:35 STEP 1291 H=1291:00 +CPU= 34.853
  23:47:29 STEP 1292 H=1292:00 +CPU= 35.609
  23:48:43 STEP 1293 H=1293:00 +CPU= 53.203
  23:49:33 STEP 1294 H=1294:00 +CPU= 35.368
  23:50:23 STEP 1295 H=1295:00 +CPU= 35.846
  23:52:14 STEP 1296 H=1296:00 +CPU= 64.079
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001296
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001296
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001296
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001272
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001272
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001272
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMGGhq0f+001284
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMSHhq0f+001284
Adding to the zip: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768/ICMUAhq0f+001284
Zipping up the intermediate file: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781_53.zip
Uploading the intermediate file: upload_file_53.zip
  23:53:04 STEP 1297 H=1297:00 +CPU= 34.788
  23:53:51 STEP 1298 H=1298:00 +CPU= 34.672
  23:55:08 STEP 1299 H=1299:00 +CPU= 54.277
  23:55:57 STEP 1300 H=1300:00 +CPU= 35.660
  23:56:42 STEP 1301 H=1301:00 +CPU= 34.466
  23:57:54 STEP 1302 H=1302:00 +CPU= 54.466
  23:58:39 STEP 1303 H=1303:00 +CPU= 35.031
  23:59:24 STEP 1304 H=1304:00 +CPU= 34.801
  00:00:34 STEP 1305 H=1305:00 +CPU= 53.913
  00:01:21 STEP 1306 H=1306:00 +CPU= 35.680
  00:02:05 STEP 1307 H=1307:00 +CPU= 33.875
  00:03:24 STEP 1308 H=1308:00 +CPU= 59.357
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMGGhq0f+001308
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMSHhq0f+001308
Moving to projects directory: /mnt/Cache/Linux/boinc-client/slots/12/ICMUAhq0f+001308
  00:04:09 STEP 1309 H=1309:00 +CPU= 35.106
  00:04:56 STEP 1310 H=1310:00 +CPU= 34.315
  00:06:05 STEP 1311 H=1311:00 +CPU= 51.612
  00:06:49 STEP 1312 H=1312:00 +CPU= 33.684
  00:07:32 STEP 1313 H=1313:00 +CPU= 32.643
  00:08:42 STEP 1314 H=1314:00 +CPU= 52.298
  00:09:31 STEP 1315 H=1315:00 +CPU= 32.386
  00:10:14 STEP 1316 H=1316:00 +CPU= 33.428
  00:11:21 STEP 1317 H=1317:00 +CPU= 50.978
  00:12:06 STEP 1318 H=1318:00 +CPU= 33.799
  00:12:49 STEP 1319 H=1319:00 +CPU= 33.457
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2010050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0124
(argv4) batchid: 979
(argv5) wuid: 12195768
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /mnt/Cache/Linux/boinc-client/slots/12
Project directory is: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12195768
..mkdir for temp folder for results failed
Copying: /mnt/Cache/Linux/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_7396b335c278c8c9540a86cf57f851ca to: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_0124_2010050100_123_979_12195768.zip
Unzipping the namelist zip file: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_ps_0124_2010050100_123_979_12195768.zip
ic_ancil_file: ic_ancil_12195768
ifsdata_file: ifsdata_12195768
climate_data_file: clim_data_12195768
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_70c068e9b28cb2c4e30c20c238cca379 to: /mnt/Cache/Linux/boinc-client/slots/12/ic_ancil_12195768.zip
Unzipping the IC ancils zip file: /mnt/Cache/Linux/boinc-client/slots/12/ic_ancil_12195768.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_f69ab474dd020aa7c8fc3c8e760a8a1f to: /mnt/Cache/Linux/boinc-client/slots/12/ifsdata/ifsdata_12195768.zip
Unzipping the ifsdata_zip file: /mnt/Cache/Linux/boinc-client/slots/12/ifsdata/ifsdata_12195768.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /mnt/Cache/Linux/boinc-client/slots/12/159l_2/clim_data_12195768.zip
Unzipping the climate data zip file: /mnt/Cache/Linux/boinc-client/slots/12/159l_2/clim_data_12195768.zip
Checking for progress XML file: /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
Opened progress file ok : /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
-- Model is restarting --
Adjusting last_iter, 1319, to previous model restart step.
Creating progress file: /mnt/Cache/Linux/boinc-client/slots/12/progress_file_12195768.xml
last_cpu_time: 55697
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0124_2010050100_123_979_12195768_0_r1449954781
The child process has been launched with process id: 83643
Executing the command: /mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.002] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.003] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.004] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001542:1674368142:0.004] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
 ABORT!    1 !! *** WAVE MODEL HAS ABORTED 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3520 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        WVCOUPLE 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         WAVEMDL 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          INITMDL 
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:92.995] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           USERIN 
 SDL_TRACEBACK: Calling INTEL_TRBK, THRD =            1
 Process            1  thread            1  calling linux_trbk from intel_trbk()
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_model.exe' : sigcontextptr=0x7ffc35fa72a0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with addr2line-cmd
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:302] /usr/bin/addr2line -fs -e '/mnt/Cache/Linux/boinc-client/slots/12/oifs_43r3_model.exe' 0x839273 0x839879 0x8c2957 0x81d89e 0x15f8aca 0x15d51c8 0x166b07f 0x15d7ff0 0x8711e0 0x7e0165 0x55e4ad 0x5471c9 0x546412 0x545e28 0x40708d 0x40220f 0x4021a2 0x1dca390 0x40206e
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [0]: [0x839273] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [1]: [0x839879] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [2]: [0x8c2957] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [3]: [0x81d89e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [4]: [0x15f8aca] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [5]: [0x15d51c8] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [6]: [0x166b07f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [7]: [0x15d7ff0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [8]: [0x8711e0] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [9]: [0x7e0165] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [10]: [0x55e4ad] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [11]: [0x5471c9] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [12]: [0x546412] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [13]: [0x545e28] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [14]: [0x40708d] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [15]: [0x40220f] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [16]: [0x4021a2] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [17]: [0x1dca390] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:319] [18]: [0x40206e] : ??() at ??:0
  [LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
 SDL_TRACEBACK: Done INTEL_TRBK, THRD =            1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:93.046] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3520MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:93.046] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:93.046] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:93.046] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x80000146bb], nsigs = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001715:1674368235:93.046] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001716:1674368236:94.048] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:Leo-15:1:1:83643:83643] [20230122:001716:1674368236:94.052] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC903B  Unknown               Unknown  Unknown
oifs_43r3_model.e  00000000008C2880  sdl_mod_mp_sdl_sr         100  sdl_mod.F90
oifs_43r3_model.e  000000000081D8AF  abor1_                     41  abor1.F90
oifs_43r3_model.e  00000000015F8ACA  mpabort_                   50  mpabort.F
oifs_43r3_model.e  00000000015D51C8  abort1_                    44  abort1.F
oifs_43r3_model.e  000000000166B07F  userin_                   553  userin.F
oifs_43r3_model.e  00000000015D7FF0  initmdl_                  452  initmdl.F
oifs_43r3_model.e  00000000008711E0  wavemdl_                  393  wavemdl.F
oifs_43r3_model.e  00000000007E0165  wvcouple_                 260  wvcouple.F90
oifs_43r3_model.e  000000000055E4AD  cnt4_                    1283  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:33:58 1507869 22281711 oifs_43r3_ps_0124_2010050100_123_979_12195768_0 2,764,800 31,939 0.0116
09 May 2023 16:33:37 1507869 22281711 oifs_43r3_ps_0124_2010050100_123_979_12195768_0 1,900,800 22,279 0.0117
09 May 2023 16:33:20 1507869 22281711 oifs_43r3_ps_0124_2010050100_123_979_12195768_0 1,036,800 12,628 0.0122


©2024 cpdn.org