<core_client_version>8.0.2</core_client_version>
<![CDATA[
<stderr_txt>
re/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001200
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001200
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001200
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001212
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001212
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001212
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001224
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001224
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001224
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001236
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001236
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001236
Zipping up the intermediate file: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_12.zip
Uploading the intermediate file: upload_file_12.zip
15:35:34 STEP 1249 H= 312:15 +CPU= 18.008
15:36:04 STEP 1250 H= 312:30 +CPU= 18.046
15:36:34 STEP 1251 H= 312:45 +CPU= 18.109
15:37:08 STEP 1252 H= 313:00 +CPU= 18.670
15:37:30 STEP 1253 H= 313:15 +CPU= 17.238
15:37:49 STEP 1254 H= 313:30 +CPU= 16.962
15:38:17 STEP 1255 H= 313:45 +CPU= 17.689
15:38:47 STEP 1256 H= 314:00 +CPU= 18.238
15:39:16 STEP 1257 H= 314:15 +CPU= 17.698
15:39:43 STEP 1258 H= 314:30 +CPU= 17.508
15:40:11 STEP 1259 H= 314:45 +CPU= 17.370
15:40:46 STEP 1260 H= 315:00 +CPU= 22.261
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001260
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001260
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001260
15:41:15 STEP 1261 H= 315:15 +CPU= 17.399
15:41:44 STEP 1262 H= 315:30 +CPU= 17.575
15:42:13 STEP 1263 H= 315:45 +CPU= 17.606
15:42:40 STEP 1264 H= 316:00 +CPU= 18.048
15:43:09 STEP 1265 H= 316:15 +CPU= 17.517
15:43:37 STEP 1266 H= 316:30 +CPU= 17.461
15:44:06 STEP 1267 H= 316:45 +CPU= 17.633
15:44:35 STEP 1268 H= 317:00 +CPU= 18.088
15:45:04 STEP 1269 H= 317:15 +CPU= 17.740
15:45:33 STEP 1270 H= 317:30 +CPU= 17.818
15:46:02 STEP 1271 H= 317:45 +CPU= 17.798
15:46:38 STEP 1272 H= 318:00 +CPU= 22.216
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001272
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001272
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001272
15:47:06 STEP 1273 H= 318:15 +CPU= 17.555
15:47:34 STEP 1274 H= 318:30 +CPU= 17.402
15:48:02 STEP 1275 H= 318:45 +CPU= 17.427
15:48:31 STEP 1276 H= 319:00 +CPU= 18.034
15:49:00 STEP 1277 H= 319:15 +CPU= 17.634
15:49:28 STEP 1278 H= 319:30 +CPU= 17.489
15:49:57 STEP 1279 H= 319:45 +CPU= 17.705
15:50:27 STEP 1280 H= 320:00 +CPU= 18.388
15:50:56 STEP 1281 H= 320:15 +CPU= 17.826
15:51:25 STEP 1282 H= 320:30 +CPU= 17.787
15:51:54 STEP 1283 H= 320:45 +CPU= 18.064
15:52:30 STEP 1284 H= 321:00 +CPU= 22.727
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001284
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001284
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001284
15:52:59 STEP 1285 H= 321:15 +CPU= 17.537
15:53:26 STEP 1286 H= 321:30 +CPU= 17.254
15:53:55 STEP 1287 H= 321:45 +CPU= 17.446
15:54:24 STEP 1288 H= 322:00 +CPU= 18.110
15:54:54 STEP 1289 H= 322:15 +CPU= 17.738
15:55:22 STEP 1290 H= 322:30 +CPU= 17.591
15:55:51 STEP 1291 H= 322:45 +CPU= 17.633
15:56:20 STEP 1292 H= 323:00 +CPU= 18.011
15:56:48 STEP 1293 H= 323:15 +CPU= 17.436
15:57:16 STEP 1294 H= 323:30 +CPU= 17.494
15:57:46 STEP 1295 H= 323:45 +CPU= 17.836
15:58:28 STEP 1296 H= 324:00 +CPU= 25.189
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001296
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001296
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:51:36 STEP 1297 H= 324:15 +CPU= 17.963
16:52:07 STEP 1298 H= 324:30 +CPU= 18.003
16:52:37 STEP 1299 H= 324:45 +CPU= 17.824
16:53:04 STEP 1300 H= 325:00 +CPU= 18.256
16:53:32 STEP 1301 H= 325:15 +CPU= 17.516
16:54:00 STEP 1302 H= 325:30 +CPU= 17.592
16:54:30 STEP 1303 H= 325:45 +CPU= 17.535
16:54:59 STEP 1304 H= 326:00 +CPU= 18.053
16:55:27 STEP 1305 H= 326:15 +CPU= 17.482
16:55:57 STEP 1306 H= 326:30 +CPU= 17.719
16:56:26 STEP 1307 H= 326:45 +CPU= 17.651
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:11:15 STEP 1308 H= 327:00 +CPU= 22.852
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001308
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001308
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001308
17:11:46 STEP 1309 H= 327:15 +CPU= 18.014
17:12:16 STEP 1310 H= 327:30 +CPU= 17.828
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:25:11 STEP 1311 H= 327:45 +CPU= 17.591
17:25:40 STEP 1312 H= 328:00 +CPU= 18.144
17:26:09 STEP 1313 H= 328:15 +CPU= 17.823
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:29:27 STEP 1314 H= 328:30 +CPU= 17.753
17:29:55 STEP 1315 H= 328:45 +CPU= 17.941
17:30:26 STEP 1316 H= 329:00 +CPU= 18.424
17:30:55 STEP 1317 H= 329:15 +CPU= 17.628
17:31:23 STEP 1318 H= 329:30 +CPU= 17.576
17:31:52 STEP 1319 H= 329:45 +CPU= 17.725
17:32:29 STEP 1320 H= 330:00 +CPU= 23.049
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001320
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001320
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001320
17:32:59 STEP 1321 H= 330:15 +CPU= 17.666
17:33:27 STEP 1322 H= 330:30 +CPU= 17.922
17:33:56 STEP 1323 H= 330:45 +CPU= 17.726
17:34:25 STEP 1324 H= 331:00 +CPU= 18.343
17:34:53 STEP 1325 H= 331:15 +CPU= 17.717
17:35:22 STEP 1326 H= 331:30 +CPU= 17.742
17:35:49 STEP 1327 H= 331:45 +CPU= 17.655
17:36:19 STEP 1328 H= 332:00 +CPU= 18.403
17:36:48 STEP 1329 H= 332:15 +CPU= 17.951
17:37:16 STEP 1330 H= 332:30 +CPU= 17.903
17:37:45 STEP 1331 H= 332:45 +CPU= 17.868
17:38:22 STEP 1332 H= 333:00 +CPU= 22.882
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001332
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001332
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001332
17:38:53 STEP 1333 H= 333:15 +CPU= 17.977
17:39:22 STEP 1334 H= 333:30 +CPU= 17.923
17:39:52 STEP 1335 H= 333:45 +CPU= 17.815
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:44:51 STEP 1336 H= 334:00 +CPU= 18.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
17:47:15 STEP 1337 H= 334:15 +CPU= 17.663
17:47:44 STEP 1338 H= 334:30 +CPU= 17.524
17:48:11 STEP 1339 H= 334:45 +CPU= 17.793
17:48:40 STEP 1340 H= 335:00 +CPU= 18.190
17:49:09 STEP 1341 H= 335:15 +CPU= 17.848
17:49:39 STEP 1342 H= 335:30 +CPU= 17.919
17:50:09 STEP 1343 H= 335:45 +CPU= 18.309
17:50:51 STEP 1344 H= 336:00 +CPU= 25.604
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001344
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001344
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001248
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001260
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001272
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001284
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001296
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001308
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001320
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001332
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001332
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001332
Zipping up the intermediate file: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_13.zip
Uploading the intermediate file: upload_file_13.zip
17:51:23 STEP 1345 H= 336:15 +CPU= 18.193
17:51:52 STEP 1346 H= 336:30 +CPU= 18.141
17:52:22 STEP 1347 H= 336:45 +CPU= 18.202
17:52:52 STEP 1348 H= 337:00 +CPU= 18.600
17:53:22 STEP 1349 H= 337:15 +CPU= 18.025
17:53:50 STEP 1350 H= 337:30 +CPU= 17.861
17:54:18 STEP 1351 H= 337:45 +CPU= 17.553
17:54:47 STEP 1352 H= 338:00 +CPU= 17.964
17:55:17 STEP 1353 H= 338:15 +CPU= 17.627
17:55:45 STEP 1354 H= 338:30 +CPU= 17.528
17:56:15 STEP 1355 H= 338:45 +CPU= 17.562
17:56:50 STEP 1356 H= 339:00 +CPU= 22.163
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001356
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001356
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001356
17:57:20 STEP 1357 H= 339:15 +CPU= 17.716
17:57:49 STEP 1358 H= 339:30 +CPU= 17.726
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
18:20:35 STEP 1359 H= 339:45 +CPU= 16.827
18:20:52 STEP 1360 H= 340:00 +CPU= 17.050
18:21:09 STEP 1361 H= 340:15 +CPU= 16.825
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:44:23 STEP 1362 H= 340:30 +CPU= 17.050
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:47:23 STEP 1363 H= 340:45 +CPU= 17.983
20:47:42 STEP 1364 H= 341:00 +CPU= 18.445
20:47:59 STEP 1365 H= 341:15 +CPU= 17.032
20:48:15 STEP 1366 H= 341:30 +CPU= 16.025
20:48:31 STEP 1367 H= 341:45 +CPU= 15.271
20:48:52 STEP 1368 H= 342:00 +CPU= 20.990
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001368
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001368
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001368
20:49:08 STEP 1369 H= 342:15 +CPU= 16.528
20:49:25 STEP 1370 H= 342:30 +CPU= 16.805
20:49:41 STEP 1371 H= 342:45 +CPU= 16.243
20:49:58 STEP 1372 H= 343:00 +CPU= 16.264
20:50:13 STEP 1373 H= 343:15 +CPU= 15.504
20:50:30 STEP 1374 H= 343:30 +CPU= 16.605
20:50:46 STEP 1375 H= 343:45 +CPU= 16.413
20:51:04 STEP 1376 H= 344:00 +CPU= 16.947
20:51:21 STEP 1377 H= 344:15 +CPU= 16.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
20:53:55 STEP 1378 H= 344:30 +CPU= 16.149
20:54:12 STEP 1379 H= 344:45 +CPU= 16.276
20:54:33 STEP 1380 H= 345:00 +CPU= 21.327
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001380
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001380
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001380
20:54:50 STEP 1381 H= 345:15 +CPU= 16.274
20:55:06 STEP 1382 H= 345:30 +CPU= 16.296
20:55:23 STEP 1383 H= 345:45 +CPU= 16.450
20:55:40 STEP 1384 H= 346:00 +CPU= 17.057
20:55:57 STEP 1385 H= 346:15 +CPU= 16.850
20:56:13 STEP 1386 H= 346:30 +CPU= 15.816
20:56:30 STEP 1387 H= 346:45 +CPU= 16.590
20:56:47 STEP 1388 H= 347:00 +CPU= 17.101
20:57:04 STEP 1389 H= 347:15 +CPU= 16.527
20:57:21 STEP 1390 H= 347:30 +CPU= 16.621
20:57:38 STEP 1391 H= 347:45 +CPU= 17.182
20:58:02 STEP 1392 H= 348:00 +CPU= 23.420
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001392
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001392
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001392
20:58:19 STEP 1393 H= 348:15 +CPU= 16.629
20:58:35 STEP 1394 H= 348:30 +CPU= 16.345
20:58:53 STEP 1395 H= 348:45 +CPU= 17.486
20:59:11 STEP 1396 H= 349:00 +CPU= 17.760
20:59:27 STEP 1397 H= 349:15 +CPU= 16.735
20:59:44 STEP 1398 H= 349:30 +CPU= 16.623
21:00:01 STEP 1399 H= 349:45 +CPU= 16.767
21:00:19 STEP 1400 H= 350:00 +CPU= 17.462
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.436] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4411MB (heap), 5565MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.449] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.449] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.449] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.449] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.449] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 4411 MB (maxheap), 5565 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS (1001)
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CLOUD_LAYER
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210028:1717894828:58629.464] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CLOUDSC
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210029:1717894829:58630.465] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:185896:185896] [20240608:210029:1717894829:58630.465] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CF935B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008214C1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DD5490 Unknown Unknown Unknown
oifs_43r3_model.e 00000000011699D4 cloudsc_ 1073 cloudsc.F90
oifs_43r3_model.e 000000000115C847 cloud_layer_ 131 cloud_layer.F90
oifs_43r3_model.e 0000000000FBB064 callpar_ 1223 callpar.F90
oifs_43r3_model.e 0000000000F5FA6B ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000F4DBC4 ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BD7131 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012C5002 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BACBAE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 0000000000688153 stepo_ 327 stepo.F90
oifs_43r3_model.e 00000000004CD1DC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000004B5D85 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 00000000004B4F52 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 000000000049D638 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 0000000000406FBD cnt0_ 146 cnt0.F90
oifs_43r3_model.e 0000000000401E7F MAIN__ 96 master.F90
oifs_43r3_model.e 0000000000401E12 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCC34A Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCDBE7 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000401CD5 Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a370
(argv4) batchid: 1017
(argv5) wuid: 12285967
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /more/var/lib/boinc-client/slots/17
Project directory is: /more/var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967
..mkdir for temp folder for results failed
Copying: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5222c0414b3b164292b8c0bc55d3aa24 to: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_a370_2016092300_20_1017_12285967.zip
Unzipping the namelist zip file: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_a370_2016092300_20_1017_12285967.zip
ic_ancil_file: ic_ancil_12285967
ifsdata_file: ifsdata_12285967
climate_data_file: clim_data_12285967
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /more/var/lib/boinc-client/slots/17/ic_ancil_12285967.zip
Unzipping the IC ancils zip file: /more/var/lib/boinc-client/slots/17/ic_ancil_12285967.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1294901d0349b53b1e985291130d35dd to: /more/var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285967.zip
Unzipping the ifsdata_zip file: /more/var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285967.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /more/var/lib/boinc-client/slots/17/159l_2/clim_data_12285967.zip
Unzipping the climate data zip file: /more/var/lib/boinc-client/slots/17/159l_2/clim_data_12285967.zip
Checking for progress XML file: /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
Opened progress file ok : /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
-- Model is restarting --
Adjusting last_iter, 1400, to previous model restart step.
Creating progress file: /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
last_cpu_time: 23742
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269
The child process has been launched with process id: 214050
Executing the command: /more/var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210127:1717894887:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:01:48 STEP 1392 H= 348:00 +CPU= 17.707
21:02:05 STEP 1393 H= 348:15 +CPU= 17.014
21:02:21 STEP 1394 H= 348:30 +CPU= 16.548
21:02:38 STEP 1395 H= 348:45 +CPU= 17.055
21:02:56 STEP 1396 H= 349:00 +CPU= 17.551
21:03:13 STEP 1397 H= 349:15 +CPU= 16.778
21:03:29 STEP 1398 H= 349:30 +CPU= 15.913
21:03:44 STEP 1399 H= 349:45 +CPU= 15.553
21:04:01 STEP 1400 H= 350:00 +CPU= 16.695
21:04:16 STEP 1401 H= 350:15 +CPU= 15.520
21:04:33 STEP 1402 H= 350:30 +CPU= 16.804
21:04:51 STEP 1403 H= 350:45 +CPU= 17.898
Suspend request received from the BOINC client, suspending the child process
double free or corruption (top)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 3958MB (heap), 5314MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x820890 (old at (nil))
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3843] 3958 MB (maxheap), 5314 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : EC_PHYS_DRV
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : >OMP-PHYSICS CLDPP T/S (1002)
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : EC_PHYS
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : CALLPAR
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : TURBULENCE_LAYER
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : VDFOUTER
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : VDFMAIN
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210526:1717895126:239.408] [c_drhook_print_@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:3897] : VDFHGHTN
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210527:1717895127:240.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:chekov:1:1:214050:214050] [20240608:210527:1717895127:240.408] [signal_drhook@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1cf8da0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
oifs_43r3_model.e 0000000001CF935B Unknown Unknown Unknown
oifs_43r3_model.e 00000000008214C1 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DD5490 Unknown Unknown Unknown
oifs_43r3_model.e 000000000123168A vdfhghtn_ 580 vdfhghtn.F90
oifs_43r3_model.e 0000000001220C63 vdfmain_ 971 vdfmain.F90
oifs_43r3_model.e 0000000001207E29 vdfouter_ 739 vdfouter.F90
oifs_43r3_model.e 00000000011FDCB2 turbulence_layer_ 158 turbulence_layer.F90
oifs_43r3_model.e 0000000000FAC003 callpar_ 869 callpar.F90
oifs_43r3_model.e 0000000000F5FA6B ec_phys_ 670 ec_phys.F90
oifs_43r3_model.e 0000000000F4DBC4 ec_phys_drv_ 599 ec_phys_drv.F90
oifs_43r3_model.e 0000000000BD7131 gp_model_ 613 gp_model.F90
oifs_43r3_model.e 00000000012C5002 gp_model_heap_ 74 gp_model_heap.F90
oifs_43r3_model.e 0000000000BACBAE scan2m_ 535 scan2m.F90
oifs_43r3_model.e 0000000000688153 stepo_ 327 stepo.F90
oifs_43r3_model.e 00000000004CD1DC cnt4_ 1133 cnt4.F90
oifs_43r3_model.e 00000000004B5D85 cnt3_ 267 cnt3.F90
oifs_43r3_model.e 00000000004B4F52 cnt2_ 88 cnt2.F90
oifs_43r3_model.e 000000000049D638 cnt1_ 92 cnt1.F90
oifs_43r3_model.e 0000000000406FBD cnt0_ 146 cnt0.F90
oifs_43r3_model.e 0000000000401E7F MAIN__ 96 master.F90
oifs_43r3_model.e 0000000000401E12 Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCC34A Unknown Unknown Unknown
oifs_43r3_model.e 0000000001DCDBE7 Unknown Unknown Unknown
oifs_43r3_model.e 0000000000401CD5 Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_bl_1.13_x86_64-pc-linux-gnu
(argv1) start_date: 2016092300
(argv2) exptid: h7zg
(argv3) unique_member_id: a370
(argv4) batchid: 1017
(argv5) wuid: 12285967
(argv6) fclen: 20
(argv7) app_name: oifs_43r3_bl
(argv8) nthreads: 1
Working directory is: /more/var/lib/boinc-client/slots/17
Project directory is: /more/var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_bl
version: 1.13
Location of temp folder: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967
..mkdir for temp folder for results failed
Copying: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip to: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_app_1.13_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5222c0414b3b164292b8c0bc55d3aa24 to: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_a370_2016092300_20_1017_12285967.zip
Unzipping the namelist zip file: /more/var/lib/boinc-client/slots/17/oifs_43r3_bl_a370_2016092300_20_1017_12285967.zip
ic_ancil_file: ic_ancil_12285967
ifsdata_file: ifsdata_12285967
climate_data_file: clim_data_12285967
horiz_resolution: 159
vert_resolution: 91
grid_type: l_2
upload_interval: 96
utstep: 900
nfrres: restart dump frequency (steps) 48
Copying IC ancils from: ../../projects/climateprediction.net/jf_f7ecfd27aa1e68bbe7e8d19800e25c73 to: /more/var/lib/boinc-client/slots/17/ic_ancil_12285967.zip
Unzipping the IC ancils zip file: /more/var/lib/boinc-client/slots/17/ic_ancil_12285967.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1294901d0349b53b1e985291130d35dd to: /more/var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285967.zip
Unzipping the ifsdata_zip file: /more/var/lib/boinc-client/slots/17/ifsdata/ifsdata_12285967.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /more/var/lib/boinc-client/slots/17/159l_2/clim_data_12285967.zip
Unzipping the climate data zip file: /more/var/lib/boinc-client/slots/17/159l_2/clim_data_12285967.zip
Checking for progress XML file: /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
Opened progress file ok : /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
-- Model is restarting --
Adjusting last_iter, 1404, to previous model restart step.
Creating progress file: /more/var/lib/boinc-client/slots/17/progress_file_12285967.xml
last_cpu_time: 23961
upload_file_number: 14
last_iter: 1393
last_upload: 1209600
model_completed: 0
total_length_of_simulation: 1728000
result_base_name: oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269
The child process has been launched with process id: 3272
Executing the command: /more/var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2091] fp = 0x314f440
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [process_options@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [ignore_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [restore_default_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [trapfpe_treatment@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x820cf0 (old at 0x1cf8da0)
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [signal_drhook_init@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x820cf0 (old at (nil))
[EC_DRHOOK:chekov:1:1:3272:3272] [20240608:211353:1717895633:0.000] [catch_signals@/home/abowery/Working_folder/OpenIFS/oifs_43r3_bl/gc_oifs43r3_2/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
21:15:10 STEP 1392 H= 348:00 +CPU= 17.499
21:15:27 STEP 1393 H= 348:15 +CPU= 16.438
21:15:44 STEP 1394 H= 348:30 +CPU= 16.676
21:16:02 STEP 1395 H= 348:45 +CPU= 16.416
21:16:20 STEP 1396 H= 349:00 +CPU= 17.324
21:16:37 STEP 1397 H= 349:15 +CPU= 16.559
21:16:55 STEP 1398 H= 349:30 +CPU= 16.745
21:17:12 STEP 1399 H= 349:45 +CPU= 16.892
21:17:31 STEP 1400 H= 350:00 +CPU= 18.064
21:17:49 STEP 1401 H= 350:15 +CPU= 16.965
21:18:07 STEP 1402 H= 350:30 +CPU= 17.440
21:18:25 STEP 1403 H= 350:45 +CPU= 17.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:19:58 STEP 1404 H= 351:00 +CPU= 22.129
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001404
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001404
21:20:16 STEP 1405 H= 351:15 +CPU= 17.195
21:20:34 STEP 1406 H= 351:30 +CPU= 17.272
21:20:52 STEP 1407 H= 351:45 +CPU= 16.598
21:21:09 STEP 1408 H= 352:00 +CPU= 16.905
21:21:27 STEP 1409 H= 352:15 +CPU= 16.716
21:21:44 STEP 1410 H= 352:30 +CPU= 16.786
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:23:04 STEP 1411 H= 352:45 +CPU= 16.482
21:23:19 STEP 1412 H= 353:00 +CPU= 15.805
21:23:35 STEP 1413 H= 353:15 +CPU= 15.466
21:23:50 STEP 1414 H= 353:30 +CPU= 15.358
21:24:06 STEP 1415 H= 353:45 +CPU= 15.809
21:24:27 STEP 1416 H= 354:00 +CPU= 21.443
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001416
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001416
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:25:46 STEP 1417 H= 354:15 +CPU= 16.861
21:26:03 STEP 1418 H= 354:30 +CPU= 16.471
21:26:21 STEP 1419 H= 354:45 +CPU= 16.972
21:26:40 STEP 1420 H= 355:00 +CPU= 17.655
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
21:28:00 STEP 1421 H= 355:15 +CPU= 17.004
21:28:17 STEP 1422 H= 355:30 +CPU= 16.797
21:28:34 STEP 1423 H= 355:45 +CPU= 16.535
21:28:52 STEP 1424 H= 356:00 +CPU= 16.959
21:29:09 STEP 1425 H= 356:15 +CPU= 16.607
21:29:27 STEP 1426 H= 356:30 +CPU= 16.673
21:29:44 STEP 1427 H= 356:45 +CPU= 16.390
21:30:07 STEP 1428 H= 357:00 +CPU= 21.965
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001428
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001428
21:30:25 STEP 1429 H= 357:15 +CPU= 16.996
21:30:43 STEP 1430 H= 357:30 +CPU= 16.794
21:31:00 STEP 1431 H= 357:45 +CPU= 16.712
21:31:18 STEP 1432 H= 358:00 +CPU= 17.549
21:31:37 STEP 1433 H= 358:15 +CPU= 17.106
21:31:55 STEP 1434 H= 358:30 +CPU= 17.161
21:32:13 STEP 1435 H= 358:45 +CPU= 17.277
21:32:32 STEP 1436 H= 359:00 +CPU= 17.982
21:32:50 STEP 1437 H= 359:15 +CPU= 17.299
21:33:09 STEP 1438 H= 359:30 +CPU= 18.134
21:33:27 STEP 1439 H= 359:45 +CPU= 17.405
21:33:50 STEP 1440 H= 360:00 +CPU= 21.531
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01
DDH------------------------------------------------------------------------
I I
I TIME = 360.000H DDH * Diagnostic Var., AREA GLOBAL I
I I
I-------------------------------------------------------------------------I
I VARIABLE I Initial value I Current value I
I-------------------------------------------------------------------------I
I OMEGA (P/S) I -0.1536505336E-20 I 0.6945032799E-05 I
ICloud fractionI 0.0000000000E+00 I 0.3048766156E-01 I
I Relat. Hum. I 0.2283444069E+00 I 0.3330349281E+00 I
I PASS 01 I 0.0000000000E+00 I 0.5029588180E+03 I
DDH------------------------------------------------------------------------
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
MAXGPFV : MAX. VALUE = 0.000000000000000E+000
NSTEP = 1440 SCAN2M_HPOS P
IO-STREAM SETUP - IOTYPE = 2 NUMIOPROCS = 1 CPATH =
ICMGGh7zg+001440 MODE=a
IO-STREAM CLOSED - ICMGGh7zg+001440
MPI-TASK: 1 - 322617764 BYTES IN 7 RECORDS TRANSFERRED IN 0.0005 SECONDS645235.5 Mbytes/s, TOTAL TIME= 0.0077( 6.5%)
21:33:50 STEP 1440 H= 360:00 +CPU= 21.531
END CNT3
NSTEP = 1440 CNT0 000000000 15.336 15.336 15.880 0.966 0 0 0 0 0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ 46/ 1069125708 WRITTEN 1430/************
-TASK-OPENED-OPEN-RECS IN -KBYTE IN -RECS OUT-KBYTE OUT-WALL -WALL IN-WALL
OU-TOT IN -TOT OUT
1 28 0 46 1044068 1430 ************ 59.2 57.4 1.8 57.7 3.6
===-=== START OF TIMING STATISTICS ===-===
STATS FOR ALL TASKS
NUM ROUTINE CALLS MEAN(ms) MAX(ms) FRAC(%) UNBAL(%)
0 CNT0 - COMPLETE EXECUTION 1 1196451.8 1196451.8 100.00 0.00
1 CNT4 - FORWARD INTEGRATION 1 1138031.0 1138031.0 95.12 0.00
8 SCAN2M - GRID-POINT DYNAMICS 53 3677.7 3677.7 16.29 0.00
9 SPCM - SPECTRAL COMP. 48 478.7 478.7 1.92 0.00
10 SCAN2M - PHYSICS 49 13194.5 13194.5 54.04 0.00
11 IOPACK - OUTPUT P.P. RESULTS 4 672.9 672.9 0.22 0.00
12 SPNORM - SPECTRAL NORM COMP. 3 23.2 23.2 0.01 0.00
17 GRIDFPOS IN CNT4 4 18.3 18.3 0.01 0.00
27 RESTART FILES - WRITING 1 2691.6 2691.6 0.22 0.00
28 RESTART FILES - READING 1 58106.6 58106.6 4.86 0.00
29 SU4FPOS IN CNT4 4 0.0 0.0 0.00 0.00
30 DYNFPOS IN CNT4 4 4138.9 4138.9 1.38 0.00
31 POSDDH IN STEPO 1 0.7 0.7 0.00 0.00
37 CPGLAG - SL COMPUTATIONS 49 3582.8 3582.8 14.67 0.00
39 SU0YOMB 1 197.7 197.7 0.02 0.00
51 SCAN2M - SL COMM. PART 1 49 13.1 13.1 0.05 0.00
54 SPCM - M TO S/S TO M TRANSP. 48 147.0 147.0 0.59 0.00
55 SPCIMPF - S TO M/M TO S TRANSP. 48 29.9 29.9 0.12 0.00
56 SPNORM - SPECTRAL NORM COMM. 3 0.1 0.1 0.00 0.00
102 LTINV_CTL - INVERSE LEGENDRE TRANSFORM 108 259.9 259.9 2.35 0.00
103 LTDIR_CTL - DIRECT LEGENDRE TRANSFORM 100 166.4 166.4 1.39 0.00
106 FTDIR_CTL - DIRECT FOURIER TRANSFORM 100 58.4 58.4 0.49 0.00
107 FTINV_CTL - INVERSE FOURIER TRANSFORM 108 95.4 95.4 0.86 0.00
140 SULEG - COMP. OF LEGENDRE POL. 1 29.1 29.1 0.00 0.00
152 LTINV_CTL - M TO L TRANSPOSITION 108 18.6 18.6 0.17 0.00
153 LTDIR_CTL - L TO M TRANSPOSITION 100 11.4 11.4 0.10 0.00
157 FTINV_CTL - L TO G TRANSPOSITION 108 48.6 48.6 0.44 0.00
158 FTDIR_CTL - G TO L TRANSPOSITION 100 25.7 25.7 0.22 0.00
400 GSTATS 6667 0.0 0.0 0.00 0.00
401 GSTATS HOOK 6149 0.0 0.0 0.00 0.00
TOTAL MEASURED IMBALANCE = 0.0 SECONDS, 0.0 PERCENT
TOTAL WALLCLOCK TIME 1196.5 CPU TIME 849.2 VECTOR TIME 849.2
===-=== END OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY 1138.8
*** END CNT0 ***
------------------------------------------------
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMGGh7zg+001440
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMSHh7zg+001440
Moving to projects directory: /more/var/lib/boinc-client/slots/17/ICMUAh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/slots/17/NODE.001_01
Adding to the zip: /more/var/lib/boinc-client/slots/17/ifs.stat
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001344
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001392
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001368
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001428
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMUAh7zg+001416
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001380
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001356
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMSHh7zg+001440
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001404
Adding to the zip: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_12285967/ICMGGh7zg+001440
Zipping up the final file: /more/var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_14.zip
Uploading the final file: upload_file_14.zip
Uploading trickle at timestep: 1295100
21:37:20 (3220): called boinc_finish(0)
</stderr_txt>
<message>
upload failure: <file_xfer_error>
<file_name>oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_15.zip</file_name>
<error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
<file_name>oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_16.zip</file_name>
<error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
<file_name>oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_17.zip</file_name>
<error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
<file_name>oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_18.zip</file_name>
<error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
<file_name>oifs_43r3_bl_a370_2016092300_20_1017_12285967_1_r526420269_19.zip</file_name>
<error_code>-161 (not found)</error_code>
</file_xfe
