climateprediction.net home page
Task 22434427

Task 22434427

Name oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0
Workunit 12282858
Created 7 Jun 2024, 12:20:15 UTC
Sent 7 Jun 2024, 15:16:09 UTC
Report deadline 6 Aug 2024, 15:16:09 UTC
Received 8 Jun 2024, 2:53:39 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1499708
Run time 7 hours 42 min 9 sec
CPU time 7 hours 35 min 35 sec
Validate state Invalid
Credit 1,976.32
Device peak FLOPS 6.71 GFLOPS
Application version OpenIFS 43r3 Baroclinic Lifecycle v1.13
x86_64-pc-linux-gnu
Peak working set size 5,455.76 MB
Peak swap size 5,980.80 MB
Peak disk usage 1,285.97 MB
Stderr
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
x/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:27525:27525] [20240608:012205:1717802525:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:22:47 STEP 1104 H= 276:00 +CPU= 19.522
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001104
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001104
  01:23:06 STEP 1105 H= 276:15 +CPU= 17.373
  01:23:24 STEP 1106 H= 276:30 +CPU= 17.250
  01:23:42 STEP 1107 H= 276:45 +CPU= 17.046
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:24:21 STEP 1108 H= 277:00 +CPU= 17.840
  01:24:40 STEP 1109 H= 277:15 +CPU= 17.880
  01:24:59 STEP 1110 H= 277:30 +CPU= 17.474
  01:25:17 STEP 1111 H= 277:45 +CPU= 17.704
  01:25:36 STEP 1112 H= 278:00 +CPU= 18.084
  01:25:54 STEP 1113 H= 278:15 +CPU= 17.187
  01:26:13 STEP 1114 H= 278:30 +CPU= 17.378
  01:26:31 STEP 1115 H= 278:45 +CPU= 17.137
  01:26:54 STEP 1116 H= 279:00 +CPU= 22.593
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001116
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001116
  01:27:12 STEP 1117 H= 279:15 +CPU= 16.885
  01:27:30 STEP 1118 H= 279:30 +CPU= 17.612
  01:27:49 STEP 1119 H= 279:45 +CPU= 17.817
  01:28:08 STEP 1120 H= 280:00 +CPU= 17.657
  01:28:26 STEP 1121 H= 280:15 +CPU= 17.381
  01:28:44 STEP 1122 H= 280:30 +CPU= 17.260
  01:29:02 STEP 1123 H= 280:45 +CPU= 17.327
  01:29:21 STEP 1124 H= 281:00 +CPU= 18.498
  01:29:40 STEP 1125 H= 281:15 +CPU= 17.610
  01:29:59 STEP 1126 H= 281:30 +CPU= 17.894
  01:30:17 STEP 1127 H= 281:45 +CPU= 17.340
  01:30:41 STEP 1128 H= 282:00 +CPU= 22.682
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001128
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001128
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:31:21 STEP 1129 H= 282:15 +CPU= 17.542
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:32:00 STEP 1130 H= 282:30 +CPU= 16.672
  01:32:19 STEP 1131 H= 282:45 +CPU= 18.257
  01:32:36 STEP 1132 H= 283:00 +CPU= 17.227
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:33:35 STEP 1133 H= 283:15 +CPU= 16.396
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:34:12 STEP 1134 H= 283:30 +CPU= 15.684
  01:34:28 STEP 1135 H= 283:45 +CPU= 15.813
  01:34:44 STEP 1136 H= 284:00 +CPU= 16.477
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:35:22 STEP 1137 H= 284:15 +CPU= 16.818
  01:35:40 STEP 1138 H= 284:30 +CPU= 17.156
  01:35:58 STEP 1139 H= 284:45 +CPU= 17.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:37:02 STEP 1140 H= 285:00 +CPU= 21.903
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001140
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001140
  01:37:21 STEP 1141 H= 285:15 +CPU= 17.443
  01:37:39 STEP 1142 H= 285:30 +CPU= 17.727
  01:37:58 STEP 1143 H= 285:45 +CPU= 17.698
  01:38:17 STEP 1144 H= 286:00 +CPU= 18.118
  01:38:35 STEP 1145 H= 286:15 +CPU= 17.353
  01:38:53 STEP 1146 H= 286:30 +CPU= 17.495
  01:39:11 STEP 1147 H= 286:45 +CPU= 17.518
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:39:51 STEP 1148 H= 287:00 +CPU= 17.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:40:32 STEP 1149 H= 287:15 +CPU= 19.175
  01:40:52 STEP 1150 H= 287:30 +CPU= 19.610
  01:41:12 STEP 1151 H= 287:45 +CPU= 19.063
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:42:00 STEP 1152 H= 288:00 +CPU= 25.092
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001152
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001056
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001068
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001080
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001092
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001104
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001116
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001128
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001140
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001140
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_11.zip
Uploading the intermediate file: upload_file_11.zip
  01:42:19 STEP 1153 H= 288:15 +CPU= 17.616
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:42:51 STEP 1154 H= 288:30 +CPU= 14.533
  01:43:11 STEP 1155 H= 288:45 +CPU= 19.472
  01:43:32 STEP 1156 H= 289:00 +CPU= 20.616
  01:43:51 STEP 1157 H= 289:15 +CPU= 18.562
  01:44:10 STEP 1158 H= 289:30 +CPU= 18.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:44:49 STEP 1159 H= 289:45 +CPU= 17.485
  01:45:09 STEP 1160 H= 290:00 +CPU= 18.704
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:45:48 STEP 1161 H= 290:15 +CPU= 17.503
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:46:26 STEP 1162 H= 290:30 +CPU= 16.815
  01:46:46 STEP 1163 H= 290:45 +CPU= 19.221
  01:47:10 STEP 1164 H= 291:00 +CPU= 23.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001164
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001164
  01:47:49 STEP 1165 H= 291:15 +CPU= 17.471
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:48:27 STEP 1166 H= 291:30 +CPU= 16.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:49:24 STEP 1167 H= 291:45 +CPU= 15.061
  01:49:42 STEP 1168 H= 292:00 +CPU= 18.111
  01:50:01 STEP 1169 H= 292:15 +CPU= 18.487
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:50:41 STEP 1170 H= 292:30 +CPU= 17.854
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:51:40 STEP 1171 H= 292:45 +CPU= 16.313
  01:51:57 STEP 1172 H= 293:00 +CPU= 16.174
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:52:54 STEP 1173 H= 293:15 +CPU= 15.276
  01:53:11 STEP 1174 H= 293:30 +CPU= 17.108
  01:53:29 STEP 1175 H= 293:45 +CPU= 16.988
  01:53:50 STEP 1176 H= 294:00 +CPU= 21.454
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001176
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:54:28 STEP 1177 H= 294:15 +CPU= 16.347
  01:54:49 STEP 1178 H= 294:30 +CPU= 19.533
  01:55:08 STEP 1179 H= 294:45 +CPU= 18.547
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:55:50 STEP 1180 H= 295:00 +CPU= 18.726
  01:56:09 STEP 1181 H= 295:15 +CPU= 18.622
  01:56:30 STEP 1182 H= 295:30 +CPU= 18.821
  01:56:50 STEP 1183 H= 295:45 +CPU= 19.014
  01:57:10 STEP 1184 H= 296:00 +CPU= 19.045
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:58:08 STEP 1185 H= 296:15 +CPU= 15.758
  01:58:27 STEP 1186 H= 296:30 +CPU= 18.475
  01:58:44 STEP 1187 H= 296:45 +CPU= 17.238
  01:59:07 STEP 1188 H= 297:00 +CPU= 22.528
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001188
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001188
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:59:46 STEP 1189 H= 297:15 +CPU= 16.756
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:00:23 STEP 1190 H= 297:30 +CPU= 16.209
  02:00:42 STEP 1191 H= 297:45 +CPU= 18.091
  02:01:02 STEP 1192 H= 298:00 +CPU= 19.678
  02:01:23 STEP 1193 H= 298:15 +CPU= 19.067
  02:01:42 STEP 1194 H= 298:30 +CPU= 18.585
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:02:44 STEP 1195 H= 298:45 +CPU= 18.169
  02:03:03 STEP 1196 H= 299:00 +CPU= 18.655
  02:03:21 STEP 1197 H= 299:15 +CPU= 17.970
  02:03:39 STEP 1198 H= 299:30 +CPU= 17.781
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:04:18 STEP 1199 H= 299:45 +CPU= 17.231
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:05:03 STEP 1200 H= 300:00 +CPU= 23.221
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001200
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001200
  02:05:21 STEP 1201 H= 300:15 +CPU= 18.059
  02:05:39 STEP 1202 H= 300:30 +CPU= 17.408
  02:05:55 STEP 1203 H= 300:45 +CPU= 16.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:06:36 STEP 1204 H= 301:00 +CPU= 18.611
  02:06:55 STEP 1205 H= 301:15 +CPU= 17.957
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:07:34 STEP 1206 H= 301:30 +CPU= 16.889
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:08:12 STEP 1207 H= 301:45 +CPU= 16.943
  02:08:33 STEP 1208 H= 302:00 +CPU= 20.412
  02:08:53 STEP 1209 H= 302:15 +CPU= 18.685
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:09:52 STEP 1210 H= 302:30 +CPU= 16.560
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:10:30 STEP 1211 H= 302:45 +CPU= 17.570
  02:10:54 STEP 1212 H= 303:00 +CPU= 22.696
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001212
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001212
  02:11:11 STEP 1213 H= 303:15 +CPU= 17.228
  02:11:29 STEP 1214 H= 303:30 +CPU= 17.588
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:12:08 STEP 1215 H= 303:45 +CPU= 17.184
  02:12:27 STEP 1216 H= 304:00 +CPU= 17.929
  02:12:45 STEP 1217 H= 304:15 +CPU= 17.106
  02:13:03 STEP 1218 H= 304:30 +CPU= 17.268
  02:13:22 STEP 1219 H= 304:45 +CPU= 17.410
  02:13:40 STEP 1220 H= 305:00 +CPU= 17.872
  02:14:00 STEP 1221 H= 305:15 +CPU= 17.731
  02:14:18 STEP 1222 H= 305:30 +CPU= 17.230
  02:14:36 STEP 1223 H= 305:45 +CPU= 17.402
  02:14:59 STEP 1224 H= 306:00 +CPU= 21.996
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001224
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001224
  02:15:17 STEP 1225 H= 306:15 +CPU= 17.468
  02:15:36 STEP 1226 H= 306:30 +CPU= 17.373
  02:15:54 STEP 1227 H= 306:45 +CPU= 17.497
  02:16:14 STEP 1228 H= 307:00 +CPU= 18.499
  02:16:32 STEP 1229 H= 307:15 +CPU= 17.985
  02:16:52 STEP 1230 H= 307:30 +CPU= 17.247
  02:17:11 STEP 1231 H= 307:45 +CPU= 17.575
  02:17:30 STEP 1232 H= 308:00 +CPU= 18.100
  02:17:49 STEP 1233 H= 308:15 +CPU= 17.482
  02:18:08 STEP 1234 H= 308:30 +CPU= 17.714
  02:18:26 STEP 1235 H= 308:45 +CPU= 17.341
  02:18:50 STEP 1236 H= 309:00 +CPU= 22.603
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001236
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001236
  02:19:08 STEP 1237 H= 309:15 +CPU= 17.390
  02:19:26 STEP 1238 H= 309:30 +CPU= 16.955
  02:19:44 STEP 1239 H= 309:45 +CPU= 17.532
  02:20:02 STEP 1240 H= 310:00 +CPU= 17.365
  02:20:21 STEP 1241 H= 310:15 +CPU= 17.495
  02:20:39 STEP 1242 H= 310:30 +CPU= 17.277
  02:20:57 STEP 1243 H= 310:45 +CPU= 17.348
  02:21:16 STEP 1244 H= 311:00 +CPU= 17.996
  02:21:34 STEP 1245 H= 311:15 +CPU= 17.462
  02:21:53 STEP 1246 H= 311:30 +CPU= 17.503
  02:22:11 STEP 1247 H= 311:45 +CPU= 17.508
  02:22:37 STEP 1248 H= 312:00 +CPU= 24.387
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001248
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001152
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001164
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001176
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001188
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001200
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001212
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001224
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001236
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001236
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_12.zip
Uploading the intermediate file: upload_file_12.zip
  02:22:55 STEP 1249 H= 312:15 +CPU= 17.363
  02:23:14 STEP 1250 H= 312:30 +CPU= 17.456
  02:23:33 STEP 1251 H= 312:45 +CPU= 17.731
  02:23:52 STEP 1252 H= 313:00 +CPU= 18.166
  02:24:10 STEP 1253 H= 313:15 +CPU= 17.419
  02:24:29 STEP 1254 H= 313:30 +CPU= 17.761
  02:24:48 STEP 1255 H= 313:45 +CPU= 17.287
  02:25:06 STEP 1256 H= 314:00 +CPU= 17.834
  02:25:25 STEP 1257 H= 314:15 +CPU= 17.325
  02:25:43 STEP 1258 H= 314:30 +CPU= 17.375
  02:26:01 STEP 1259 H= 314:45 +CPU= 17.501
  02:26:25 STEP 1260 H= 315:00 +CPU= 22.317
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001260
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001260
  02:26:44 STEP 1261 H= 315:15 +CPU= 17.699
  02:27:02 STEP 1262 H= 315:30 +CPU= 17.520
  02:27:21 STEP 1263 H= 315:45 +CPU= 17.603
  02:27:40 STEP 1264 H= 316:00 +CPU= 18.380
  02:27:58 STEP 1265 H= 316:15 +CPU= 17.085
  02:28:18 STEP 1266 H= 316:30 +CPU= 18.012
  02:28:36 STEP 1267 H= 316:45 +CPU= 17.325
  02:28:55 STEP 1268 H= 317:00 +CPU= 18.035
  02:29:14 STEP 1269 H= 317:15 +CPU= 17.429
  02:29:33 STEP 1270 H= 317:30 +CPU= 17.730
  02:29:51 STEP 1271 H= 317:45 +CPU= 17.621
  02:30:15 STEP 1272 H= 318:00 +CPU= 22.538
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001272
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001272
  02:30:34 STEP 1273 H= 318:15 +CPU= 17.486
  02:30:52 STEP 1274 H= 318:30 +CPU= 17.294
  02:31:11 STEP 1275 H= 318:45 +CPU= 17.446
  02:31:30 STEP 1276 H= 319:00 +CPU= 17.628
  02:31:48 STEP 1277 H= 319:15 +CPU= 17.824
  02:32:06 STEP 1278 H= 319:30 +CPU= 17.240
  02:32:24 STEP 1279 H= 319:45 +CPU= 17.231
  02:32:43 STEP 1280 H= 320:00 +CPU= 18.054
  02:33:01 STEP 1281 H= 320:15 +CPU= 17.261
  02:33:20 STEP 1282 H= 320:30 +CPU= 17.552
  02:33:38 STEP 1283 H= 320:45 +CPU= 17.737
  02:34:02 STEP 1284 H= 321:00 +CPU= 22.012
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001284
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001284
  02:34:21 STEP 1285 H= 321:15 +CPU= 17.473
  02:34:39 STEP 1286 H= 321:30 +CPU= 17.085
  02:34:57 STEP 1287 H= 321:45 +CPU= 17.531
  02:35:16 STEP 1288 H= 322:00 +CPU= 17.945
  02:35:35 STEP 1289 H= 322:15 +CPU= 17.552
  02:35:53 STEP 1290 H= 322:30 +CPU= 17.478
  02:36:11 STEP 1291 H= 322:45 +CPU= 17.334
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:36:52 STEP 1292 H= 323:00 +CPU= 18.401
  02:37:11 STEP 1293 H= 323:15 +CPU= 17.618
  02:37:29 STEP 1294 H= 323:30 +CPU= 17.330
  02:37:48 STEP 1295 H= 323:45 +CPU= 17.751
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a0sn
(argv4) batchid: 1017
(argv5) wuid: 12282858
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/12
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/12/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_2cdb90c0c64f71f07e8ad471a77dbd24 to: /var/lib/boinc/slots/12/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/12/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858.zip
ic_ancil_file: ic_ancil_12282858
ifsdata_file: ifsdata_12282858
climate_data_file: clim_data_12282858
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /var/lib/boinc/slots/12/ic_ancil_12282858.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/12/ic_ancil_12282858.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_60a52b0e9d7db12cc77bb2ab773ab633 to: /var/lib/boinc/slots/12/ifsdata/ifsdata_12282858.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/12/ifsdata/ifsdata_12282858.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/12/159l_2/clim_data_12282858.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/12/159l_2/clim_data_12282858.zip
Checking for progress XML file: /var/lib/boinc/slots/12/progress_file_12282858.xml
Opened progress file ok : /var/lib/boinc/slots/12/progress_file_12282858.xml
-- Model is restarting --
Adjusting last_iter, 1296, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/12/progress_file_12282858.xml
last_cpu_time: 24738
upload_file_number: 13
last_iter: 1297
last_upload: 1123200
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035
The child process has been launched with process id: 28715
Executing the command: /var/lib/boinc/slots/12/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:etam-komp:1:1:28715:28715] [20240608:024805:1717807685:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:48:48 STEP 1296 H= 324:00 +CPU= 18.839
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001296
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001296
  02:49:07 STEP 1297 H= 324:15 +CPU= 17.980
  02:49:25 STEP 1298 H= 324:30 +CPU= 17.793
  02:49:43 STEP 1299 H= 324:45 +CPU= 17.552
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:50:23 STEP 1300 H= 325:00 +CPU= 17.849
  02:50:41 STEP 1301 H= 325:15 +CPU= 17.502
  02:51:00 STEP 1302 H= 325:30 +CPU= 17.618
  02:51:18 STEP 1303 H= 325:45 +CPU= 17.431
  02:51:37 STEP 1304 H= 326:00 +CPU= 18.041
  02:51:55 STEP 1305 H= 326:15 +CPU= 17.515
  02:52:13 STEP 1306 H= 326:30 +CPU= 17.346
  02:52:32 STEP 1307 H= 326:45 +CPU= 17.268
  02:52:55 STEP 1308 H= 327:00 +CPU= 22.263
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001308
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001308
  02:53:13 STEP 1309 H= 327:15 +CPU= 17.311
  02:53:31 STEP 1310 H= 327:30 +CPU= 17.212
  02:53:50 STEP 1311 H= 327:45 +CPU= 17.656
  02:54:09 STEP 1312 H= 328:00 +CPU= 18.066
  02:54:28 STEP 1313 H= 328:15 +CPU= 17.840
  02:54:46 STEP 1314 H= 328:30 +CPU= 17.076
  02:55:05 STEP 1315 H= 328:45 +CPU= 17.638
  02:55:22 STEP 1316 H= 329:00 +CPU= 17.126
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:56:02 STEP 1317 H= 329:15 +CPU= 17.620
  02:56:21 STEP 1318 H= 329:30 +CPU= 18.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:57:03 STEP 1319 H= 329:45 +CPU= 19.819
  02:57:29 STEP 1320 H= 330:00 +CPU= 24.107
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001320
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001320
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:58:06 STEP 1321 H= 330:15 +CPU= 16.518
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  02:58:45 STEP 1322 H= 330:30 +CPU= 17.281
  02:59:02 STEP 1323 H= 330:45 +CPU= 16.806
  02:59:19 STEP 1324 H= 331:00 +CPU= 17.015
  02:59:36 STEP 1325 H= 331:15 +CPU= 16.281
  02:59:53 STEP 1326 H= 331:30 +CPU= 17.097
  03:00:12 STEP 1327 H= 331:45 +CPU= 18.102
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:00:52 STEP 1328 H= 332:00 +CPU= 17.719
  03:01:10 STEP 1329 H= 332:15 +CPU= 18.061
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:01:49 STEP 1330 H= 332:30 +CPU= 17.384
  03:02:08 STEP 1331 H= 332:45 +CPU= 18.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:02:53 STEP 1332 H= 333:00 +CPU= 23.350
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001332
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001332
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:03:32 STEP 1333 H= 333:15 +CPU= 17.481
  03:03:48 STEP 1334 H= 333:30 +CPU= 15.988
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:04:25 STEP 1335 H= 333:45 +CPU= 15.089
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:05:02 STEP 1336 H= 334:00 +CPU= 16.256
  03:05:18 STEP 1337 H= 334:15 +CPU= 15.548
  03:05:35 STEP 1338 H= 334:30 +CPU= 16.869
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:06:14 STEP 1339 H= 334:45 +CPU= 17.644
  03:06:32 STEP 1340 H= 335:00 +CPU= 17.963
  03:06:49 STEP 1341 H= 335:15 +CPU= 16.401
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:07:30 STEP 1342 H= 335:30 +CPU= 18.182
  03:07:50 STEP 1343 H= 335:45 +CPU= 19.620
  03:08:18 STEP 1344 H= 336:00 +CPU= 26.697
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001344
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001248
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001260
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001272
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001284
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001296
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001308
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001320
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001332
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001332
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_13.zip
Uploading the intermediate file: upload_file_13.zip
  03:09:00 STEP 1345 H= 336:15 +CPU= 19.057
  03:09:20 STEP 1346 H= 336:30 +CPU= 18.861
  03:09:39 STEP 1347 H= 336:45 +CPU= 18.259
  03:09:58 STEP 1348 H= 337:00 +CPU= 18.650
  03:10:16 STEP 1349 H= 337:15 +CPU= 17.980
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:11:15 STEP 1350 H= 337:30 +CPU= 15.941
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:11:52 STEP 1351 H= 337:45 +CPU= 16.335
  03:12:11 STEP 1352 H= 338:00 +CPU= 18.222
  03:12:29 STEP 1353 H= 338:15 +CPU= 17.463
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:13:06 STEP 1354 H= 338:30 +CPU= 15.758
  03:13:21 STEP 1355 H= 338:45 +CPU= 15.680
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:14:06 STEP 1356 H= 339:00 +CPU= 22.650
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001356
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001356
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:14:45 STEP 1357 H= 339:15 +CPU= 17.374
  03:15:04 STEP 1358 H= 339:30 +CPU= 18.455
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:15:42 STEP 1359 H= 339:45 +CPU= 17.325
  03:15:59 STEP 1360 H= 340:00 +CPU= 16.886
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:16:37 STEP 1361 H= 340:15 +CPU= 16.406
  03:16:53 STEP 1362 H= 340:30 +CPU= 15.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:17:31 STEP 1363 H= 340:45 +CPU= 16.615
  03:17:50 STEP 1364 H= 341:00 +CPU= 18.099
  03:18:07 STEP 1365 H= 341:15 +CPU= 16.929
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:18:45 STEP 1366 H= 341:30 +CPU= 17.085
  03:19:03 STEP 1367 H= 341:45 +CPU= 17.318
  03:19:25 STEP 1368 H= 342:00 +CPU= 22.025
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001368
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001368
  03:19:44 STEP 1369 H= 342:15 +CPU= 17.926
  03:20:02 STEP 1370 H= 342:30 +CPU= 17.819
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:20:40 STEP 1371 H= 342:45 +CPU= 16.443
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:21:18 STEP 1372 H= 343:00 +CPU= 16.550
  03:21:37 STEP 1373 H= 343:15 +CPU= 18.106
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:22:15 STEP 1374 H= 343:30 +CPU= 16.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:22:55 STEP 1375 H= 343:45 +CPU= 17.403
  03:23:14 STEP 1376 H= 344:00 +CPU= 17.908
  03:23:32 STEP 1377 H= 344:15 +CPU= 17.694
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:24:12 STEP 1378 H= 344:30 +CPU= 17.752
  03:24:30 STEP 1379 H= 344:45 +CPU= 17.561
  03:24:54 STEP 1380 H= 345:00 +CPU= 22.332
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001380
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001380
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:25:33 STEP 1381 H= 345:15 +CPU= 17.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:26:12 STEP 1382 H= 345:30 +CPU= 17.634
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:26:49 STEP 1383 H= 345:45 +CPU= 15.489
  03:27:08 STEP 1384 H= 346:00 +CPU= 18.583
  03:27:26 STEP 1385 H= 346:15 +CPU= 17.514
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:28:05 STEP 1386 H= 346:30 +CPU= 17.461
  03:28:23 STEP 1387 H= 346:45 +CPU= 17.037
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:29:23 STEP 1388 H= 347:00 +CPU= 18.796
  03:29:43 STEP 1389 H= 347:15 +CPU= 18.671
  03:30:02 STEP 1390 H= 347:30 +CPU= 18.399
  03:30:21 STEP 1391 H= 347:45 +CPU= 18.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:31:08 STEP 1392 H= 348:00 +CPU= 24.593
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001392
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001392
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:31:45 STEP 1393 H= 348:15 +CPU= 16.394
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:32:24 STEP 1394 H= 348:30 +CPU= 17.410
  03:32:43 STEP 1395 H= 348:45 +CPU= 18.313
  03:33:04 STEP 1396 H= 349:00 +CPU= 19.115
  03:33:23 STEP 1397 H= 349:15 +CPU= 18.205
  03:33:42 STEP 1398 H= 349:30 +CPU= 18.135
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:34:40 STEP 1399 H= 349:45 +CPU= 15.898
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:35:17 STEP 1400 H= 350:00 +CPU= 15.704
  03:35:33 STEP 1401 H= 350:15 +CPU= 15.983
  03:35:49 STEP 1402 H= 350:30 +CPU= 16.313
  03:36:06 STEP 1403 H= 350:45 +CPU= 16.269
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:36:50 STEP 1404 H= 351:00 +CPU= 22.741
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001404
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001404
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:37:49 STEP 1405 H= 351:15 +CPU= 15.726
  03:38:07 STEP 1406 H= 351:30 +CPU= 17.413
  03:38:24 STEP 1407 H= 351:45 +CPU= 16.908
  03:38:42 STEP 1408 H= 352:00 +CPU= 17.583
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:39:20 STEP 1409 H= 352:15 +CPU= 16.859
  03:39:37 STEP 1410 H= 352:30 +CPU= 17.027
  03:39:55 STEP 1411 H= 352:45 +CPU= 17.344
  03:40:13 STEP 1412 H= 353:00 +CPU= 17.373
  03:40:31 STEP 1413 H= 353:15 +CPU= 17.300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  03:41:09 STEP 1414 H= 353:30 +CPU= 17.335
  03:41:29 STEP 1415 H= 353:45 +CPU= 17.810
  03:41:52 STEP 1416 H= 354:00 +CPU= 22.565
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001416
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001416
  03:42:11 STEP 1417 H= 354:15 +CPU= 17.575
  03:42:29 STEP 1418 H= 354:30 +CPU= 17.569
  03:42:47 STEP 1419 H= 354:45 +CPU= 17.149
  03:43:06 STEP 1420 H= 355:00 +CPU= 17.906
  03:43:25 STEP 1421 H= 355:15 +CPU= 17.580
  03:43:43 STEP 1422 H= 355:30 +CPU= 17.268
  03:44:01 STEP 1423 H= 355:45 +CPU= 17.399
  03:44:20 STEP 1424 H= 356:00 +CPU= 18.099
  03:44:39 STEP 1425 H= 356:15 +CPU= 17.581
  03:44:57 STEP 1426 H= 356:30 +CPU= 17.760
  03:45:15 STEP 1427 H= 356:45 +CPU= 17.389
  03:45:40 STEP 1428 H= 357:00 +CPU= 22.758
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001428
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001428
  03:45:59 STEP 1429 H= 357:15 +CPU= 17.863
  03:46:18 STEP 1430 H= 357:30 +CPU= 17.693
  03:46:37 STEP 1431 H= 357:45 +CPU= 17.421
  03:46:56 STEP 1432 H= 358:00 +CPU= 18.143
  03:47:14 STEP 1433 H= 358:15 +CPU= 17.436
  03:47:32 STEP 1434 H= 358:30 +CPU= 17.253
  03:47:50 STEP 1435 H= 358:45 +CPU= 17.334
  03:48:08 STEP 1436 H= 359:00 +CPU= 17.449
  03:48:27 STEP 1437 H= 359:15 +CPU= 17.400
  03:48:45 STEP 1438 H= 359:30 +CPU= 17.319
  03:49:04 STEP 1439 H= 359:45 +CPU= 17.482
  03:49:27 STEP 1440 H= 360:00 +CPU= 22.742
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 
DDH------------------------------------------------------------------------
I                                                                         I
I TIME =  360.000H DDH * Diagnostic Var.,     AREA                GLOBAL  I
I                                                                         I
I-------------------------------------------------------------------------I
I   VARIABLE   I      Initial value         I        Current value        I
I-------------------------------------------------------------------------I
I  OMEGA (P/S) I     -0.6055118847E-21      I       0.3264728270E-06      I
ICloud fractionI      0.0000000000E+00      I       0.6535022530E-02      I
I  Relat. Hum. I      0.4088409784E+00      I       0.4280099460E+00      I
I  PASS 01     I      0.0000000000E+00      I       0.1223222780E+03      I
DDH------------------------------------------------------------------------
  
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
  MAXGPFV : MAX. VALUE =   0.000000000000000E+000
 NSTEP =  1440 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGh7zg+001440 MODE=a
 IO-STREAM CLOSED - ICMGGh7zg+001440
 MPI-TASK:   1 -    843908084 BYTES IN      7 RECORDS TRANSFERRED IN    0.0004 SECONDS******** Mbytes/s, TOTAL TIME=    0.0073( 5.5%)
 03:49:27 STEP 1440 H= 360:00 +CPU= 22.742
 END CNT3
 NSTEP =  1440 CNT0   000000000  16.597  16.597  17.475   0.950               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           46/  1069125708 WRITTEN         4290/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     78    0       46      1044068     4290 ************     5.5     1.1     4.4     1.6     9.7
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3682023.7 3682023.7    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3679474.9 3679474.9     99.93      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  157    3663.4    3663.4     15.62      0.00
   9 SPCM     - SPECTRAL COMP.                     144     988.5     988.5      3.87      0.00
  10 SCAN2M - PHYSICS                              145   12606.5   12606.5     49.65      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 12     517.0     517.0      0.17      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                  7      29.5      29.5      0.01      0.00
  17 GRIDFPOS IN CNT4                               12      22.4      22.4      0.01      0.00
  27 RESTART FILES - WRITING                         3    2064.2    2064.2      0.17      0.00
  28 RESTART FILES - READING                         1    2163.0    2163.0      0.06      0.00
  29 SU4FPOS IN CNT4                                12       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                12    4322.1    4322.1      1.41      0.00
  31 POSDDH IN STEPO                                 1       1.2       1.2      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    145    4959.5    4959.5     19.53      0.00
  39 SU0YOMB                                         1     253.3     253.3      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    145      14.7      14.7      0.06      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              144     205.5     205.5      0.80      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              144      35.7      35.7      0.14      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                  7       0.2       0.2      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      316     390.8     390.8      3.35      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       300     171.0     171.0      1.39      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        300      60.0      60.0      0.49      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       316     100.6     100.6      0.86      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            1      33.6      33.6      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            316      23.4      23.4      0.20      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            300      84.3      84.3      0.69      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            316     190.9     190.9      1.64      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            300      30.6      30.6      0.25      0.00
 400 GSTATS                                      17763       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 16473       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3682.0 CPU TIME    2597.9 VECTOR TIME     2597.9


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY      352.2
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/12/ICMGGh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/12/ICMSHh7zg+001440
Moving to projects directory: /var/lib/boinc/slots/12/ICMUAh7zg+001440
Adding to the zip: /var/lib/boinc/slots/12/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/12/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001344
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001356
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001368
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001380
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001392
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001404
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001416
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001428
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMGGh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMSHh7zg+001440
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_12282858/ICMUAh7zg+001440
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
03:52:56 (28702): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_15.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_16.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_17.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_18.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0_r312673035_19.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_e
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
08 Jun 2024 01:51:51 1499708 22434427 oifs_43r3_bl_a0sn_2016092300_20_1017_12282858_0 1,295,100 27,335 0.0211


©2024 cpdn.org