climateprediction.net home page
Task 22261927

Task 22261927

Name oifs_43r3_ps_0504_1991050100_123_960_12177148_0
Workunit 12177148
Created 21 Dec 2022, 21:59:03 UTC
Sent 11 Jan 2023, 21:13:25 UTC
Report deadline 10 Feb 2023, 21:13:25 UTC
Received 14 Jan 2023, 4:25:37 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1519251
Run time 6 hours 31 min 17 sec
CPU time 5 hours 5 min 10 sec
Validate state Invalid
Credit 0.00
Device peak FLOPS 4.94 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,417.86 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,676.24 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
lient, suspending the child process
Resuming the child process
  05:32:06 STEP  450 H= 450:00 +CPU= 34.269
  05:32:26 STEP  451 H= 451:00 +CPU= 20.292
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:35:40 STEP  452 H= 452:00 +CPU= 21.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:38:44 STEP  453 H= 453:00 +CPU= 32.951
  05:39:05 STEP  454 H= 454:00 +CPU= 20.932
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:41:49 STEP  455 H= 455:00 +CPU= 21.551
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:45:01 STEP  456 H= 456:00 +CPU= 39.061
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000456
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000432
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000444
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000444
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_18.zip
Uploading the intermediate file: upload_file_18.zip
  05:45:20 STEP  457 H= 457:00 +CPU= 18.804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:47:53 STEP  458 H= 458:00 +CPU= 20.145
  05:48:26 STEP  459 H= 459:00 +CPU= 33.208
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:52:20 STEP  460 H= 460:00 +CPU= 21.289
  05:52:41 STEP  461 H= 461:00 +CPU= 20.785
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:56:38 STEP  462 H= 462:00 +CPU= 34.370
  05:56:58 STEP  463 H= 463:00 +CPU= 20.001
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  05:59:40 STEP  464 H= 464:00 +CPU= 20.798
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:02:15 STEP  465 H= 465:00 +CPU= 33.286
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:03:56 STEP  466 H= 466:00 +CPU= 20.785
  06:04:16 STEP  467 H= 467:00 +CPU= 19.715
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:07:05 STEP  468 H= 468:00 +CPU= 36.737
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000468
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000468
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:09:26 STEP  469 H= 469:00 +CPU= 20.203
  06:09:48 STEP  470 H= 470:00 +CPU= 21.268
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:12:31 STEP  471 H= 471:00 +CPU= 32.341
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:14:32 STEP  472 H= 472:00 +CPU= 21.056
  06:14:53 STEP  473 H= 473:00 +CPU= 20.422
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:17:28 STEP  474 H= 474:00 +CPU= 33.539
  06:17:48 STEP  475 H= 475:00 +CPU= 20.686
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:19:40 STEP  476 H= 476:00 +CPU= 20.701
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:22:54 STEP  477 H= 477:00 +CPU= 33.105
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:24:56 STEP  478 H= 478:00 +CPU= 20.734
  06:25:18 STEP  479 H= 479:00 +CPU= 21.481
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:27:28 STEP  480 H= 480:00 +CPU= 38.675
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000480
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000456
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000468
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000468
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_19.zip
Uploading the intermediate file: upload_file_19.zip
  06:27:46 STEP  481 H= 481:00 +CPU= 17.734
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:30:01 STEP  482 H= 482:00 +CPU= 19.909
  06:30:33 STEP  483 H= 483:00 +CPU= 31.808
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:33:15 STEP  484 H= 484:00 +CPU= 21.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:35:06 STEP  485 H= 485:00 +CPU= 20.547
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:36:50 STEP  486 H= 486:00 +CPU= 33.319
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:38:51 STEP  487 H= 487:00 +CPU= 20.108
  06:39:13 STEP  488 H= 488:00 +CPU= 21.015
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:42:27 STEP  489 H= 489:00 +CPU= 32.609
  06:42:49 STEP  490 H= 490:00 +CPU= 21.306
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:45:10 STEP  491 H= 491:00 +CPU= 20.313
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:48:09 STEP  492 H= 492:00 +CPU= 36.217
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000492
  06:48:29 STEP  493 H= 493:00 +CPU= 20.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0504
(argv4) batchid: 960
(argv5) wuid: 12177148
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5617b1ad626c29c7e2063fe046c4f2ac to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
ic_ancil_file: ic_ancil_12177148
ifsdata_file: ifsdata_12177148
climate_data_file: clim_data_12177148
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58a83fb67a10cf22a575ee82102fe373 to: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9397c08ef793ab7d857c0c28c4ad81ca to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12177148.xml
-- Model is restarting --
Adjusting last_iter, 494, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
last_cpu_time: 13851
upload_file_number: 20
last_iter: 479
last_upload: 1728000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209
The child process has been launched with process id: 1955152
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1955152:1955152] [20230114:065321:1673646801:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:56:01 STEP  480 H= 480:00 +CPU= 36.184
  06:56:21 STEP  481 H= 481:00 +CPU= 20.432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  06:58:33 STEP  482 H= 482:00 +CPU= 20.471
  06:59:05 STEP  483 H= 483:00 +CPU= 32.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:01:47 STEP  484 H= 484:00 +CPU= 20.765
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:05:21 STEP  485 H= 485:00 +CPU= 21.457
  07:05:55 STEP  486 H= 486:00 +CPU= 33.465
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:08:26 STEP  487 H= 487:00 +CPU= 20.486
  07:08:48 STEP  488 H= 488:00 +CPU= 21.038
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:12:32 STEP  489 H= 489:00 +CPU= 33.048
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:14:45 STEP  490 H= 490:00 +CPU= 20.743
  07:15:06 STEP  491 H= 491:00 +CPU= 21.074
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:17:34 STEP  492 H= 492:00 +CPU= 36.432
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000492
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000492
  07:20:26 STEP  493 H= 493:00 +CPU= 20.650
  07:20:47 STEP  494 H= 494:00 +CPU= 20.883
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:23:30 STEP  495 H= 495:00 +CPU= 32.444
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:26:04 STEP  496 H= 496:00 +CPU= 21.038
  07:26:26 STEP  497 H= 497:00 +CPU= 21.804
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:29:20 STEP  498 H= 498:00 +CPU= 32.819
  07:29:41 STEP  499 H= 499:00 +CPU= 21.007
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:31:43 STEP  500 H= 500:00 +CPU= 21.219
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:33:57 STEP  501 H= 501:00 +CPU= 32.090
  07:34:18 STEP  502 H= 502:00 +CPU= 21.504
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:37:11 STEP  503 H= 503:00 +CPU= 21.388
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:39:53 STEP  504 H= 504:00 +CPU= 37.924
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000504
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000480
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000492
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000492
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_20.zip
Uploading the intermediate file: upload_file_20.zip
  07:40:13 STEP  505 H= 505:00 +CPU= 20.704
  07:40:34 STEP  506 H= 506:00 +CPU= 20.843
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:43:29 STEP  507 H= 507:00 +CPU= 32.740
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:45:43 STEP  508 H= 508:00 +CPU= 21.281
  07:46:03 STEP  509 H= 509:00 +CPU= 20.451
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:49:09 STEP  510 H= 510:00 +CPU= 33.830
  07:49:30 STEP  511 H= 511:00 +CPU= 20.156
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:51:51 STEP  512 H= 512:00 +CPU= 20.803
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:54:05 STEP  513 H= 513:00 +CPU= 32.286
  07:54:27 STEP  514 H= 514:00 +CPU= 21.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  07:57:20 STEP  515 H= 515:00 +CPU= 20.761
  07:57:56 STEP  516 H= 516:00 +CPU= 36.062
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000516
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000516
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:00:28 STEP  517 H= 517:00 +CPU= 20.355
  08:00:50 STEP  518 H= 518:00 +CPU= 21.755
  08:01:22 STEP  519 H= 519:00 +CPU= 32.005
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:04:05 STEP  520 H= 520:00 +CPU= 21.129
  08:04:26 STEP  521 H= 521:00 +CPU= 20.755
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:07:10 STEP  522 H= 522:00 +CPU= 32.890
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:09:47 STEP  523 H= 523:00 +CPU= 20.171
  08:10:09 STEP  524 H= 524:00 +CPU= 21.483
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:12:33 STEP  525 H= 525:00 +CPU= 32.508
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:14:46 STEP  526 H= 526:00 +CPU= 21.090
  08:15:07 STEP  527 H= 527:00 +CPU= 21.382
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:17:28 STEP  528 H= 528:00 +CPU= 37.963
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000528
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000504
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000516
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000516
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_21.zip
Uploading the intermediate file: upload_file_21.zip
  08:17:47 STEP  529 H= 529:00 +CPU= 18.854
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:19:55 STEP  530 H= 530:00 +CPU= 19.296
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:22:19 STEP  531 H= 531:00 +CPU= 33.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:24:02 STEP  532 H= 532:00 +CPU= 21.187
  08:24:23 STEP  533 H= 533:00 +CPU= 21.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:26:48 STEP  534 H= 534:00 +CPU= 33.052
  08:27:07 STEP  535 H= 535:00 +CPU= 19.748
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:29:50 STEP  536 H= 536:00 +CPU= 21.508
  08:30:23 STEP  537 H= 537:00 +CPU= 32.269
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:33:56 STEP  538 H= 538:00 +CPU= 21.497
  08:34:16 STEP  539 H= 539:00 +CPU= 20.015
  08:34:52 STEP  540 H= 540:00 +CPU= 35.654
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000540
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000540
  08:37:04 STEP  541 H= 541:00 +CPU= 20.629
  08:37:25 STEP  542 H= 542:00 +CPU= 21.099
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:40:19 STEP  543 H= 543:00 +CPU= 32.475
  08:40:39 STEP  544 H= 544:00 +CPU= 20.297
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:43:22 STEP  545 H= 545:00 +CPU= 20.177
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:45:37 STEP  546 H= 546:00 +CPU= 33.873
  08:45:58 STEP  547 H= 547:00 +CPU= 21.010
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:48:29 STEP  548 H= 548:00 +CPU= 20.998
  08:49:03 STEP  549 H= 549:00 +CPU= 33.622
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:51:27 STEP  550 H= 550:00 +CPU= 20.929
  08:51:48 STEP  551 H= 551:00 +CPU= 20.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0504
(argv4) batchid: 960
(argv5) wuid: 12177148
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5617b1ad626c29c7e2063fe046c4f2ac to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
ic_ancil_file: ic_ancil_12177148
ifsdata_file: ifsdata_12177148
climate_data_file: clim_data_12177148
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58a83fb67a10cf22a575ee82102fe373 to: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9397c08ef793ab7d857c0c28c4ad81ca to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12177148.xml
-- Model is restarting --
Adjusting last_iter, 552, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
last_cpu_time: 15695
upload_file_number: 22
last_iter: 551
last_upload: 1900800
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209
The child process has been launched with process id: 1973319
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1973319:1973319] [20230114:085715:1673654235:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  08:59:27 STEP  552 H= 552:00 +CPU= 37.606
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000552
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000528
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000540
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000540
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_22.zip
Uploading the intermediate file: upload_file_22.zip
  08:59:47 STEP  553 H= 553:00 +CPU= 19.747
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:02:28 STEP  554 H= 554:00 +CPU= 20.043
  09:03:02 STEP  555 H= 555:00 +CPU= 33.337
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:06:14 STEP  556 H= 556:00 +CPU= 20.626
  09:06:35 STEP  557 H= 557:00 +CPU= 20.961
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:09:10 STEP  558 H= 558:00 +CPU= 33.825
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:11:12 STEP  559 H= 559:00 +CPU= 20.735
  09:11:34 STEP  560 H= 560:00 +CPU= 21.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:14:07 STEP  561 H= 561:00 +CPU= 32.677
  09:14:28 STEP  562 H= 562:00 +CPU= 20.917
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:16:32 STEP  563 H= 563:00 +CPU= 21.169
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:20:43 STEP  564 H= 564:00 +CPU= 37.122
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000564
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000564
  09:21:04 STEP  565 H= 565:00 +CPU= 21.081
  09:21:25 STEP  566 H= 566:00 +CPU= 20.896
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:24:09 STEP  567 H= 567:00 +CPU= 32.747
  09:24:30 STEP  568 H= 568:00 +CPU= 20.868
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:26:52 STEP  569 H= 569:00 +CPU= 21.013
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:29:17 STEP  570 H= 570:00 +CPU= 33.589
  09:29:37 STEP  571 H= 571:00 +CPU= 20.004
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:32:00 STEP  572 H= 572:00 +CPU= 20.933
  09:32:33 STEP  573 H= 573:00 +CPU= 32.805
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:35:05 STEP  574 H= 574:00 +CPU= 20.889
  09:35:26 STEP  575 H= 575:00 +CPU= 21.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:39:37 STEP  576 H= 576:00 +CPU= 38.252
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000576
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000552
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000564
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000564
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_23.zip
Uploading the intermediate file: upload_file_23.zip
  09:39:57 STEP  577 H= 577:00 +CPU= 20.286
  09:40:18 STEP  578 H= 578:00 +CPU= 20.824
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:44:14 STEP  579 H= 579:00 +CPU= 32.319
  09:44:35 STEP  580 H= 580:00 +CPU= 21.409
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:46:46 STEP  581 H= 581:00 +CPU= 20.215
  09:47:19 STEP  582 H= 582:00 +CPU= 33.429
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:49:51 STEP  583 H= 583:00 +CPU= 20.372
  09:50:12 STEP  584 H= 584:00 +CPU= 21.613
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:52:35 STEP  585 H= 585:00 +CPU= 31.563
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:54:39 STEP  586 H= 586:00 +CPU= 21.203
  09:54:59 STEP  587 H= 587:00 +CPU= 20.675
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  09:57:26 STEP  588 H= 588:00 +CPU= 35.818
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000588
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000588
  09:57:46 STEP  589 H= 589:00 +CPU= 19.515
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:00:08 STEP  590 H= 590:00 +CPU= 20.949
  10:00:41 STEP  591 H= 591:00 +CPU= 32.858
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:05:34 STEP  592 H= 592:00 +CPU= 21.555
  10:05:54 STEP  593 H= 593:00 +CPU= 19.977
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:08:39 STEP  594 H= 594:00 +CPU= 33.490
  10:08:58 STEP  595 H= 595:00 +CPU= 19.060
  10:09:18 STEP  596 H= 596:00 +CPU= 19.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:12:02 STEP  597 H= 597:00 +CPU= 32.359
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:14:24 STEP  598 H= 598:00 +CPU= 21.068
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:16:37 STEP  599 H= 599:00 +CPU= 21.648
  10:17:14 STEP  600 H= 600:00 +CPU= 37.600
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000600
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000576
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000588
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000588
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_24.zip
Uploading the intermediate file: upload_file_24.zip
  10:17:32 STEP  601 H= 601:00 +CPU= 17.696
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:20:14 STEP  602 H= 602:00 +CPU= 21.165
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:22:58 STEP  603 H= 603:00 +CPU= 33.123
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:25:51 STEP  604 H= 604:00 +CPU= 21.366
  10:26:12 STEP  605 H= 605:00 +CPU= 20.316
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:29:27 STEP  606 H= 606:00 +CPU= 33.637
  10:29:48 STEP  607 H= 607:00 +CPU= 21.203
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:32:11 STEP  608 H= 608:00 +CPU= 21.566
  10:32:43 STEP  609 H= 609:00 +CPU= 32.365
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:36:25 STEP  610 H= 610:00 +CPU= 21.461
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:39:28 STEP  611 H= 611:00 +CPU= 20.550
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:42:54 STEP  612 H= 612:00 +CPU= 36.366
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:45:07 STEP  613 H= 613:00 +CPU= 21.724
  10:45:30 STEP  614 H= 614:00 +CPU= 21.948
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:48:25 STEP  615 H= 615:00 +CPU= 33.670
  10:48:46 STEP  616 H= 616:00 +CPU= 21.014
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:52:18 STEP  617 H= 617:00 +CPU= 20.553
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:55:12 STEP  618 H= 618:00 +CPU= 34.311
  10:55:33 STEP  619 H= 619:00 +CPU= 20.709
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  10:58:46 STEP  620 H= 620:00 +CPU= 21.644
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0504
(argv4) batchid: 960
(argv5) wuid: 12177148
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/13
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_5617b1ad626c29c7e2063fe046c4f2ac to: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/13/oifs_43r3_ps_0504_1991050100_123_960_12177148.zip
ic_ancil_file: ic_ancil_12177148
ifsdata_file: ifsdata_12177148
climate_data_file: clim_data_12177148
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_58a83fb67a10cf22a575ee82102fe373 to: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/13/ic_ancil_12177148.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_9397c08ef793ab7d857c0c28c4ad81ca to: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/13/ifsdata/ifsdata_12177148.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/13/159l_2/clim_data_12177148.zip
Checking for progress XML file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
Opened progress file ok : /var/lib/boinc-client/slots/13/progress_file_12177148.xml
-- Model is restarting --
Adjusting last_iter, 620, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/13/progress_file_12177148.xml
last_cpu_time: 17430
upload_file_number: 25
last_iter: 599
last_upload: 2160000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209
The child process has been launched with process id: 1992714
Executing the command: /var/lib/boinc-client/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:p500:1:1:1992714:1992714] [20230114:110116:1673661676:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:04:19 STEP  600 H= 600:00 +CPU= 37.216
  11:04:40 STEP  601 H= 601:00 +CPU= 20.318
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:08:12 STEP  602 H= 602:00 +CPU= 21.228
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:11:27 STEP  603 H= 603:00 +CPU= 33.208
  11:11:48 STEP  604 H= 604:00 +CPU= 20.986
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:14:40 STEP  605 H= 605:00 +CPU= 20.573
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:17:35 STEP  606 H= 606:00 +CPU= 34.108
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:20:48 STEP  607 H= 607:00 +CPU= 20.455
  11:21:11 STEP  608 H= 608:00 +CPU= 22.541
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:27:07 STEP  609 H= 609:00 +CPU= 32.658
  11:27:28 STEP  610 H= 610:00 +CPU= 21.058
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:30:20 STEP  611 H= 611:00 +CPU= 21.032
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:34:07 STEP  612 H= 612:00 +CPU= 35.933
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000612
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000612
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:36:39 STEP  613 H= 613:00 +CPU= 20.659
  11:37:01 STEP  614 H= 614:00 +CPU= 21.527
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:42:37 STEP  615 H= 615:00 +CPU= 32.870
  11:42:59 STEP  616 H= 616:00 +CPU= 22.021
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:46:02 STEP  617 H= 617:00 +CPU= 21.150
  11:46:35 STEP  618 H= 618:00 +CPU= 33.140
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:49:37 STEP  619 H= 619:00 +CPU= 21.095
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:52:42 STEP  620 H= 620:00 +CPU= 21.892
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:55:25 STEP  621 H= 621:00 +CPU= 32.554
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  11:58:18 STEP  622 H= 622:00 +CPU= 20.921
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:00:29 STEP  623 H= 623:00 +CPU= 21.178
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:07:31 STEP  624 H= 624:00 +CPU= 39.872
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMGGhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMSHhq0f+000624
Moving to projects directory: /var/lib/boinc-client/slots/13/ICMUAhq0f+000624
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000600
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMGGhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMSHhq0f+000612
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12177148/ICMUAhq0f+000612
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0504_1991050100_123_960_12177148_0_r1102513209_25.zip
Uploading the intermediate file: upload_file_25.zip
  12:07:51 STEP  625 H= 625:00 +CPU= 19.415
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:10:26 STEP  626 H= 626:00 +CPU= 21.158
  12:10:59 STEP  627 H= 627:00 +CPU= 32.415
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:14:42 STEP  628 H= 628:00 +CPU= 21.582
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:17:14 STEP  629 H= 629:00 +CPU= 21.176
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:20:00 STEP  630 H= 630:00 +CPU= 34.171
  12:20:20 STEP  631 H= 631:00 +CPU= 20.402
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:24:43 STEP  632 H= 632:00 +CPU= 21.351
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  12:28:36 STEP  633 H= 633:00 +CPU= 31.964
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
No trickles!


©2024 climateprediction.net