<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:21:37 STEP 1435 H=1435:00 +CPU= 23.126
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:22:05 STEP 1436 H=1436:00 +CPU= 23.658
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:22:47 STEP 1437 H=1437:00 +CPU= 36.731
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:23:12 STEP 1438 H=1438:00 +CPU= 23.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:23:41 STEP 1439 H=1439:00 +CPU= 24.088
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001416
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001416
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001416
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001428
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001428
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001428
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_57.zip
Uploading the intermediate file: upload_file_57.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:24:25 STEP 1440 H=1440:00 +CPU= 40.711
Moving to projects directory: /root/slots/8/ICMGGhq0f+001440
Moving to projects directory: /root/slots/8/ICMSHhq0f+001440
Moving to projects directory: /root/slots/8/ICMUAhq0f+001440
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:24:51 STEP 1441 H=1441:00 +CPU= 23.285
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:25:18 STEP 1442 H=1442:00 +CPU= 24.193
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:26:01 STEP 1443 H=1443:00 +CPU= 36.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:26:28 STEP 1444 H=1444:00 +CPU= 24.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:26:54 STEP 1445 H=1445:00 +CPU= 23.300
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:27:36 STEP 1446 H=1446:00 +CPU= 36.564
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:28:03 STEP 1447 H=1447:00 +CPU= 23.210
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:28:30 STEP 1448 H=1448:00 +CPU= 23.539
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:29:10 STEP 1449 H=1449:00 +CPU= 35.790
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:29:37 STEP 1450 H=1450:00 +CPU= 24.353
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:30:05 STEP 1451 H=1451:00 +CPU= 22.879
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:30:49 STEP 1452 H=1452:00 +CPU= 39.355
Moving to projects directory: /root/slots/8/ICMGGhq0f+001452
Moving to projects directory: /root/slots/8/ICMSHhq0f+001452
Moving to projects directory: /root/slots/8/ICMUAhq0f+001452
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:31:14 STEP 1453 H=1453:00 +CPU= 23.037
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:31:42 STEP 1454 H=1454:00 +CPU= 23.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:32:21 STEP 1455 H=1455:00 +CPU= 34.801
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:32:47 STEP 1456 H=1456:00 +CPU= 22.692
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:33:11 STEP 1457 H=1457:00 +CPU= 22.292
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:33:52 STEP 1458 H=1458:00 +CPU= 35.880
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:34:17 STEP 1459 H=1459:00 +CPU= 22.535
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:34:45 STEP 1460 H=1460:00 +CPU= 23.329
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:35:25 STEP 1461 H=1461:00 +CPU= 35.243
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:35:51 STEP 1462 H=1462:00 +CPU= 23.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:36:16 STEP 1463 H=1463:00 +CPU= 23.498
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001440
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001440
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001440
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001452
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001452
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001452
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_58.zip
Uploading the intermediate file: upload_file_58.zip
Uploading trickle at timestep: 5266800
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:37:01 STEP 1464 H=1464:00 +CPU= 39.909
Moving to projects directory: /root/slots/8/ICMGGhq0f+001464
Moving to projects directory: /root/slots/8/ICMSHhq0f+001464
Moving to projects directory: /root/slots/8/ICMUAhq0f+001464
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:37:27 STEP 1465 H=1465:00 +CPU= 23.137
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:37:53 STEP 1466 H=1466:00 +CPU= 23.584
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:38:33 STEP 1467 H=1467:00 +CPU= 35.526
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:38:59 STEP 1468 H=1468:00 +CPU= 23.496
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:39:26 STEP 1469 H=1469:00 +CPU= 22.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:06 STEP 1470 H=1470:00 +CPU= 36.025
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:31 STEP 1471 H=1471:00 +CPU= 22.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:40:56 STEP 1472 H=1472:00 +CPU= 22.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:41:37 STEP 1473 H=1473:00 +CPU= 35.509
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:42:06 STEP 1474 H=1474:00 +CPU= 24.093
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:42:31 STEP 1475 H=1475:00 +CPU= 22.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:43:15 STEP 1476 H=1476:00 +CPU= 39.259
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/8/ICMGGhq0f+001476
Moving to projects directory: /root/slots/8/ICMSHhq0f+001476
Moving to projects directory: /root/slots/8/ICMUAhq0f+001476
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:43:42 STEP 1477 H=1477:00 +CPU= 22.229
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:44:07 STEP 1478 H=1478:00 +CPU= 22.811
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:44:46 STEP 1479 H=1479:00 +CPU= 34.954
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:45:13 STEP 1480 H=1480:00 +CPU= 23.869
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:45:38 STEP 1481 H=1481:00 +CPU= 22.596
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:46:21 STEP 1482 H=1482:00 +CPU= 36.162
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:46:46 STEP 1483 H=1483:00 +CPU= 22.700
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:47:12 STEP 1484 H=1484:00 +CPU= 23.385
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:47:53 STEP 1485 H=1485:00 +CPU= 35.922
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:48:21 STEP 1486 H=1486:00 +CPU= 23.989
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:50:19 STEP 1487 H=1487:00 +CPU= 26.834
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001464
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001464
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001464
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001476
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001476
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001476
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_59.zip
Uploading the intermediate file: upload_file_59.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:51:40 STEP 1488 H=1488:00 +CPU= 43.231
Moving to projects directory: /root/slots/8/ICMGGhq0f+001488
Moving to projects directory: /root/slots/8/ICMSHhq0f+001488
Moving to projects directory: /root/slots/8/ICMUAhq0f+001488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:52:08 STEP 1489 H=1489:00 +CPU= 25.340
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:52:37 STEP 1490 H=1490:00 +CPU= 24.794
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:53:12 STEP 1491 H=1491:00 +CPU= 30.222
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:53:34 STEP 1492 H=1492:00 +CPU= 19.720
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0492
(argv4) batchid: 957
(argv5) wuid: 12174136
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/8
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12174136
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_998817141bcf9a8f975aa28e2e89f665 to: /root/slots/8/oifs_43r3_ps_0492_1988050100_123_957_12174136.zip
Unzipping the namelist zip file: /root/slots/8/oifs_43r3_ps_0492_1988050100_123_957_12174136.zip
ic_ancil_file: ic_ancil_12174136
ifsdata_file: ifsdata_12174136
climate_data_file: clim_data_12174136
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f8151baa63a8b7d829e2c96faf3f3812 to: /root/slots/8/ic_ancil_12174136.zip
Unzipping the IC ancils zip file: /root/slots/8/ic_ancil_12174136.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_49ba63a0d0288d5380ed12917032f503 to: /root/slots/8/ifsdata/ifsdata_12174136.zip
Unzipping the ifsdata_zip file: /root/slots/8/ifsdata/ifsdata_12174136.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/8/159l_2/clim_data_12174136.zip
Unzipping the climate data zip file: /root/slots/8/159l_2/clim_data_12174136.zip
Checking for progress XML file: /root/slots/8/progress_file_12174136.xml
Opened progress file ok : /root/slots/8/progress_file_12174136.xml
-- Model is restarting --
Adjusting last_iter, 1493, to previous model restart step.
Creating progress file: /root/slots/8/progress_file_12174136.xml
last_cpu_time: 48629
upload_file_number: 60
last_iter: 1487
last_upload: 5353200
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754
The child process has been launched with process id: 11682
Executing the command: /root/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:11682:11682] [20230108:155446:1673189686:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:56:01 STEP 1488 H=1488:00 +CPU= 39.275
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:56:27 STEP 1489 H=1489:00 +CPU= 24.524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:56:56 STEP 1490 H=1490:00 +CPU= 25.791
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:57:38 STEP 1491 H=1491:00 +CPU= 37.743
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:58:08 STEP 1492 H=1492:00 +CPU= 25.260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:58:34 STEP 1493 H=1493:00 +CPU= 24.221
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:59:23 STEP 1494 H=1494:00 +CPU= 40.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
15:59:51 STEP 1495 H=1495:00 +CPU= 25.425
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:00:23 STEP 1496 H=1496:00 +CPU= 25.872
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:01:10 STEP 1497 H=1497:00 +CPU= 39.914
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:01:37 STEP 1498 H=1498:00 +CPU= 25.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:02:06 STEP 1499 H=1499:00 +CPU= 24.944
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:02:54 STEP 1500 H=1500:00 +CPU= 43.389
Moving to projects directory: /root/slots/8/ICMGGhq0f+001500
Moving to projects directory: /root/slots/8/ICMSHhq0f+001500
Moving to projects directory: /root/slots/8/ICMUAhq0f+001500
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:03:22 STEP 1501 H=1501:00 +CPU= 23.980
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:03:51 STEP 1502 H=1502:00 +CPU= 26.925
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:04:32 STEP 1503 H=1503:00 +CPU= 37.545
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:04:58 STEP 1504 H=1504:00 +CPU= 23.473
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:05:25 STEP 1505 H=1505:00 +CPU= 23.468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:06:08 STEP 1506 H=1506:00 +CPU= 38.152
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:06:35 STEP 1507 H=1507:00 +CPU= 25.593
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:07:04 STEP 1508 H=1508:00 +CPU= 24.187
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:07:47 STEP 1509 H=1509:00 +CPU= 38.864
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:08:13 STEP 1510 H=1510:00 +CPU= 24.216
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:08:42 STEP 1511 H=1511:00 +CPU= 24.697
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001488
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001488
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001488
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001500
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001500
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001500
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_60.zip
Uploading the intermediate file: upload_file_60.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:09:28 STEP 1512 H=1512:00 +CPU= 44.183
Moving to projects directory: /root/slots/8/ICMGGhq0f+001512
Moving to projects directory: /root/slots/8/ICMSHhq0f+001512
Moving to projects directory: /root/slots/8/ICMUAhq0f+001512
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:09:56 STEP 1513 H=1513:00 +CPU= 25.899
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:10:25 STEP 1514 H=1514:00 +CPU= 25.505
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:11:08 STEP 1515 H=1515:00 +CPU= 38.080
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:11:35 STEP 1516 H=1516:00 +CPU= 25.774
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:12:03 STEP 1517 H=1517:00 +CPU= 23.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:12:37 STEP 1518 H=1518:00 +CPU= 31.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:13:01 STEP 1519 H=1519:00 +CPU= 19.999
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:13:25 STEP 1520 H=1520:00 +CPU= 21.882
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:03 STEP 1521 H=1521:00 +CPU= 34.260
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:26 STEP 1522 H=1522:00 +CPU= 21.077
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:48 STEP 1523 H=1523:00 +CPU= 19.132
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:15:29 STEP 1524 H=1524:00 +CPU= 37.176
Moving to projects directory: /root/slots/8/ICMGGhq0f+001524
Moving to projects directory: /root/slots/8/ICMSHhq0f+001524
Moving to projects directory: /root/slots/8/ICMUAhq0f+001524
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:15:52 STEP 1525 H=1525:00 +CPU= 20.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:16:15 STEP 1526 H=1526:00 +CPU= 21.250
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:16:51 STEP 1527 H=1527:00 +CPU= 32.290
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:17:13 STEP 1528 H=1528:00 +CPU= 20.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:17:36 STEP 1529 H=1529:00 +CPU= 20.853
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:18:16 STEP 1530 H=1530:00 +CPU= 36.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:18:41 STEP 1531 H=1531:00 +CPU= 20.711
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:19:05 STEP 1532 H=1532:00 +CPU= 22.090
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:19:42 STEP 1533 H=1533:00 +CPU= 32.377
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:20:05 STEP 1534 H=1534:00 +CPU= 21.665
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:20:30 STEP 1535 H=1535:00 +CPU= 22.434
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001512
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001512
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001512
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001524
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001524
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001524
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_61.zip
Uploading the intermediate file: upload_file_61.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:21:09 STEP 1536 H=1536:00 +CPU= 36.527
Moving to projects directory: /root/slots/8/ICMGGhq0f+001536
Moving to projects directory: /root/slots/8/ICMSHhq0f+001536
Moving to projects directory: /root/slots/8/ICMUAhq0f+001536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:21:30 STEP 1537 H=1537:00 +CPU= 19.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:21:51 STEP 1538 H=1538:00 +CPU= 19.126
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:22:27 STEP 1539 H=1539:00 +CPU= 32.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:22:50 STEP 1540 H=1540:00 +CPU= 20.151
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:23:11 STEP 1541 H=1541:00 +CPU= 19.028
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:23:48 STEP 1542 H=1542:00 +CPU= 33.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:24:11 STEP 1543 H=1543:00 +CPU= 20.237
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:24:33 STEP 1544 H=1544:00 +CPU= 20.248
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:25:10 STEP 1545 H=1545:00 +CPU= 33.474
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:25:33 STEP 1546 H=1546:00 +CPU= 20.607
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:25:57 STEP 1547 H=1547:00 +CPU= 22.114
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:26:38 STEP 1548 H=1548:00 +CPU= 37.103
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/8/ICMGGhq0f+001548
Moving to projects directory: /root/slots/8/ICMSHhq0f+001548
Moving to projects directory: /root/slots/8/ICMUAhq0f+001548
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:27:01 STEP 1549 H=1549:00 +CPU= 19.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:27:27 STEP 1550 H=1550:00 +CPU= 23.543
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:07 STEP 1551 H=1551:00 +CPU= 35.490
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:32 STEP 1552 H=1552:00 +CPU= 22.959
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:55 STEP 1553 H=1553:00 +CPU= 21.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:29:36 STEP 1554 H=1554:00 +CPU= 37.041
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:30:02 STEP 1555 H=1555:00 +CPU= 22.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:30:27 STEP 1556 H=1556:00 +CPU= 23.158
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:31:01 STEP 1557 H=1557:00 +CPU= 30.128
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:31:24 STEP 1558 H=1558:00 +CPU= 21.040
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:31:47 STEP 1559 H=1559:00 +CPU= 20.896
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001536
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001536
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001536
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001548
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001548
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001548
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_62.zip
Uploading the intermediate file: upload_file_62.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:32:27 STEP 1560 H=1560:00 +CPU= 37.882
Moving to projects directory: /root/slots/8/ICMGGhq0f+001560
Moving to projects directory: /root/slots/8/ICMSHhq0f+001560
Moving to projects directory: /root/slots/8/ICMUAhq0f+001560
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:32:50 STEP 1561 H=1561:00 +CPU= 20.514
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:33:13 STEP 1562 H=1562:00 +CPU= 21.370
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:33:50 STEP 1563 H=1563:00 +CPU= 32.363
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:34:42 STEP 1564 H=1564:00 +CPU= 22.823
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:35:06 STEP 1565 H=1565:00 +CPU= 21.254
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:35:44 STEP 1566 H=1566:00 +CPU= 35.127
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:36:07 STEP 1567 H=1567:00 +CPU= 20.420
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:36:30 STEP 1568 H=1568:00 +CPU= 20.330
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:37:06 STEP 1569 H=1569:00 +CPU= 31.000
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:37:27 STEP 1570 H=1570:00 +CPU= 19.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:37:49 STEP 1571 H=1571:00 +CPU= 20.064
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1988050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0492
(argv4) batchid: 957
(argv5) wuid: 12174136
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /root/slots/8
Project directory is: /root/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /root/projects/climateprediction.net/oifs_43r3_ps_12174136
..mkdir for temp folder for results failed
Copying: /root/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /root/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /root/slots/8/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_998817141bcf9a8f975aa28e2e89f665 to: /root/slots/8/oifs_43r3_ps_0492_1988050100_123_957_12174136.zip
Unzipping the namelist zip file: /root/slots/8/oifs_43r3_ps_0492_1988050100_123_957_12174136.zip
ic_ancil_file: ic_ancil_12174136
ifsdata_file: ifsdata_12174136
climate_data_file: clim_data_12174136
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_f8151baa63a8b7d829e2c96faf3f3812 to: /root/slots/8/ic_ancil_12174136.zip
Unzipping the IC ancils zip file: /root/slots/8/ic_ancil_12174136.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_49ba63a0d0288d5380ed12917032f503 to: /root/slots/8/ifsdata/ifsdata_12174136.zip
Unzipping the ifsdata_zip file: /root/slots/8/ifsdata/ifsdata_12174136.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /root/slots/8/159l_2/clim_data_12174136.zip
Unzipping the climate data zip file: /root/slots/8/159l_2/clim_data_12174136.zip
Checking for progress XML file: /root/slots/8/progress_file_12174136.xml
Opened progress file ok : /root/slots/8/progress_file_12174136.xml
-- Model is restarting --
Adjusting last_iter, 1572, to previous model restart step.
Creating progress file: /root/slots/8/progress_file_12174136.xml
last_cpu_time: 50919
upload_file_number: 63
last_iter: 1559
last_upload: 5612400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754
The child process has been launched with process id: 7669
Executing the command: /root/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:52-dienstlich:1:1:7669:7669] [20230109:161343:1673277223:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:29 STEP 1560 H=1560:00 +CPU= 37.029
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:14:57 STEP 1561 H=1561:00 +CPU= 23.884
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:15:25 STEP 1562 H=1562:00 +CPU= 22.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:16:04 STEP 1563 H=1563:00 +CPU= 34.142
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:16:30 STEP 1564 H=1564:00 +CPU= 23.086
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:17:23 STEP 1565 H=1565:00 +CPU= 24.072
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:18:08 STEP 1566 H=1566:00 +CPU= 38.161
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:18:52 STEP 1567 H=1567:00 +CPU= 24.101
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:19:54 STEP 1568 H=1568:00 +CPU= 25.288
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:20:51 STEP 1569 H=1569:00 +CPU= 36.057
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:21:16 STEP 1570 H=1570:00 +CPU= 22.538
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:21:42 STEP 1571 H=1571:00 +CPU= 22.552
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:22:30 STEP 1572 H=1572:00 +CPU= 39.248
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/8/ICMGGhq0f+001572
Moving to projects directory: /root/slots/8/ICMSHhq0f+001572
Moving to projects directory: /root/slots/8/ICMUAhq0f+001572
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:23:06 STEP 1573 H=1573:00 +CPU= 22.520
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:23:52 STEP 1574 H=1574:00 +CPU= 24.912
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:24:35 STEP 1575 H=1575:00 +CPU= 36.202
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:25:18 STEP 1576 H=1576:00 +CPU= 24.991
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:26:01 STEP 1577 H=1577:00 +CPU= 25.536
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:27:01 STEP 1578 H=1578:00 +CPU= 40.491
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:27:54 STEP 1579 H=1579:00 +CPU= 25.631
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:28:38 STEP 1580 H=1580:00 +CPU= 26.339
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:29:59 STEP 1581 H=1581:00 +CPU= 40.101
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:30:42 STEP 1582 H=1582:00 +CPU= 26.273
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:31:14 STEP 1583 H=1583:00 +CPU= 26.222
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001560
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001560
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001560
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMGGhq0f+001572
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMSHhq0f+001572
Adding to the zip: /root/projects/climateprediction.net/oifs_43r3_ps_12174136/ICMUAhq0f+001572
Zipping up the intermediate file: /root/projects/climateprediction.net/oifs_43r3_ps_0492_1988050100_123_957_12174136_0_r1366719754_63.zip
Uploading the intermediate file: upload_file_63.zip
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:32:08 STEP 1584 H=1584:00 +CPU= 40.426
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /root/slots/8/ICMGGhq0f+001584
Moving to projects directory: /root/slots/8/ICMSHhq0f+001584
Moving to projects directory: /root/slots/8/ICMUAhq0f+001584
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:32:51 STEP 1585 H=1585:00 +CPU= 25.258
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
16:33:32 STEP 1586 H=1586:00 +CPU= 24.484
Abort request received from BOINC client, ending the child process
</stderr_txt>
]]>
