climateprediction.net home page
Task 22297906

Task 22297906

Name oifs_43r3_ps_0077_1989050100_123_958_12174721_2
Workunit 12174721
Created 14 Jan 2023, 18:52:02 UTC
Sent 14 Jan 2023, 18:55:45 UTC
Report deadline 13 Feb 2023, 18:55:45 UTC
Received 16 Jan 2023, 23:37:20 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1532338
Run time 21 hours 28 min 55 sec
CPU time 21 hours 22 min 32 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.63 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,802.45 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,834.19 MB
Stderr
<core_client_version>7.16.16</core_client_version>
<![CDATA[
<stderr_txt>
/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_104.zip
Uploading the intermediate file: upload_file_104.zip
  07:26:45 STEP 2521 H=2521:00 +CPU= 13.897
  07:27:00 STEP 2522 H=2522:00 +CPU= 14.403
  07:27:22 STEP 2523 H=2523:00 +CPU= 21.995
  07:27:36 STEP 2524 H=2524:00 +CPU= 13.807
  07:27:50 STEP 2525 H=2525:00 +CPU= 13.929
  07:28:13 STEP 2526 H=2526:00 +CPU= 23.407
  07:28:27 STEP 2527 H=2527:00 +CPU= 14.355
  07:28:42 STEP 2528 H=2528:00 +CPU= 14.621
  07:29:05 STEP 2529 H=2529:00 +CPU= 22.830
  07:29:19 STEP 2530 H=2530:00 +CPU= 14.487
  07:29:33 STEP 2531 H=2531:00 +CPU= 14.163
  07:29:58 STEP 2532 H=2532:00 +CPU= 24.774
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002532
  07:30:12 STEP 2533 H=2533:00 +CPU= 14.181
  07:30:27 STEP 2534 H=2534:00 +CPU= 14.561
  07:30:49 STEP 2535 H=2535:00 +CPU= 22.190
  07:31:04 STEP 2536 H=2536:00 +CPU= 14.537
  07:31:17 STEP 2537 H=2537:00 +CPU= 13.523
  07:31:40 STEP 2538 H=2538:00 +CPU= 23.064
  07:31:54 STEP 2539 H=2539:00 +CPU= 14.058
  07:32:09 STEP 2540 H=2540:00 +CPU= 14.545
  07:32:32 STEP 2541 H=2541:00 +CPU= 22.730
  07:32:45 STEP 2542 H=2542:00 +CPU= 13.749
  07:32:59 STEP 2543 H=2543:00 +CPU= 13.704
  07:33:25 STEP 2544 H=2544:00 +CPU= 26.174
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_105.zip
Uploading the intermediate file: upload_file_105.zip
  07:33:39 STEP 2545 H=2545:00 +CPU= 13.881
  07:33:53 STEP 2546 H=2546:00 +CPU= 14.059
  07:34:15 STEP 2547 H=2547:00 +CPU= 21.688
  07:34:30 STEP 2548 H=2548:00 +CPU= 14.701
  07:34:43 STEP 2549 H=2549:00 +CPU= 13.883
  07:35:06 STEP 2550 H=2550:00 +CPU= 22.134
  07:35:19 STEP 2551 H=2551:00 +CPU= 13.089
  07:35:33 STEP 2552 H=2552:00 +CPU= 14.455
  07:35:56 STEP 2553 H=2553:00 +CPU= 22.876
  07:36:10 STEP 2554 H=2554:00 +CPU= 14.005
  07:36:24 STEP 2555 H=2555:00 +CPU= 14.024
  07:36:49 STEP 2556 H=2556:00 +CPU= 24.967
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002556
  07:37:03 STEP 2557 H=2557:00 +CPU= 14.043
  07:37:18 STEP 2558 H=2558:00 +CPU= 14.646
  07:37:40 STEP 2559 H=2559:00 +CPU= 22.703
  07:37:55 STEP 2560 H=2560:00 +CPU= 14.570
  07:38:09 STEP 2561 H=2561:00 +CPU= 14.196
  07:38:32 STEP 2562 H=2562:00 +CPU= 23.281
  07:38:46 STEP 2563 H=2563:00 +CPU= 14.020
  07:39:01 STEP 2564 H=2564:00 +CPU= 14.988
  07:39:24 STEP 2565 H=2565:00 +CPU= 22.261
  07:39:38 STEP 2566 H=2566:00 +CPU= 13.888
  07:39:53 STEP 2567 H=2567:00 +CPU= 14.945
  07:40:19 STEP 2568 H=2568:00 +CPU= 26.843
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_106.zip
Uploading the intermediate file: upload_file_106.zip
  07:40:34 STEP 2569 H=2569:00 +CPU= 14.169
  07:40:48 STEP 2570 H=2570:00 +CPU= 14.390
  07:41:10 STEP 2571 H=2571:00 +CPU= 22.380
  07:41:25 STEP 2572 H=2572:00 +CPU= 14.468
  07:41:39 STEP 2573 H=2573:00 +CPU= 14.027
  07:42:02 STEP 2574 H=2574:00 +CPU= 22.981
  07:42:16 STEP 2575 H=2575:00 +CPU= 13.735
  07:42:30 STEP 2576 H=2576:00 +CPU= 14.008
  07:42:52 STEP 2577 H=2577:00 +CPU= 22.674
  07:43:06 STEP 2578 H=2578:00 +CPU= 13.919
  07:43:20 STEP 2579 H=2579:00 +CPU= 13.438
  07:43:44 STEP 2580 H=2580:00 +CPU= 23.989
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002580
  07:43:57 STEP 2581 H=2581:00 +CPU= 13.303
  07:44:11 STEP 2582 H=2582:00 +CPU= 14.222
  07:44:33 STEP 2583 H=2583:00 +CPU= 22.298
  07:44:48 STEP 2584 H=2584:00 +CPU= 14.423
  07:45:02 STEP 2585 H=2585:00 +CPU= 14.475
  07:45:26 STEP 2586 H=2586:00 +CPU= 23.222
  07:45:39 STEP 2587 H=2587:00 +CPU= 13.798
  07:45:54 STEP 2588 H=2588:00 +CPU= 14.237
  07:46:17 STEP 2589 H=2589:00 +CPU= 22.906
  07:46:31 STEP 2590 H=2590:00 +CPU= 14.032
  07:46:44 STEP 2591 H=2591:00 +CPU= 13.860
  07:47:11 STEP 2592 H=2592:00 +CPU= 26.828
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_107.zip
Uploading the intermediate file: upload_file_107.zip
  07:47:25 STEP 2593 H=2593:00 +CPU= 14.194
  07:47:40 STEP 2594 H=2594:00 +CPU= 14.352
  07:48:02 STEP 2595 H=2595:00 +CPU= 22.523
  07:48:17 STEP 2596 H=2596:00 +CPU= 14.666
  07:48:31 STEP 2597 H=2597:00 +CPU= 14.049
  07:48:54 STEP 2598 H=2598:00 +CPU= 23.218
  07:49:09 STEP 2599 H=2599:00 +CPU= 14.377
  07:49:23 STEP 2600 H=2600:00 +CPU= 14.237
  07:49:45 STEP 2601 H=2601:00 +CPU= 21.699
  07:49:59 STEP 2602 H=2602:00 +CPU= 14.509
  07:50:13 STEP 2603 H=2603:00 +CPU= 13.753
  07:50:38 STEP 2604 H=2604:00 +CPU= 24.869
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002604
  07:50:52 STEP 2605 H=2605:00 +CPU= 14.073
  07:51:06 STEP 2606 H=2606:00 +CPU= 14.502
  07:51:29 STEP 2607 H=2607:00 +CPU= 22.853
  07:51:44 STEP 2608 H=2608:00 +CPU= 14.376
  07:51:58 STEP 2609 H=2609:00 +CPU= 14.089
  07:52:21 STEP 2610 H=2610:00 +CPU= 22.964
  07:52:34 STEP 2611 H=2611:00 +CPU= 13.381
  07:52:48 STEP 2612 H=2612:00 +CPU= 13.734
  07:53:10 STEP 2613 H=2613:00 +CPU= 22.558
  07:53:24 STEP 2614 H=2614:00 +CPU= 13.537
  07:53:38 STEP 2615 H=2615:00 +CPU= 14.297
  07:54:05 STEP 2616 H=2616:00 +CPU= 26.796
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_108.zip
Uploading the intermediate file: upload_file_108.zip
  07:54:19 STEP 2617 H=2617:00 +CPU= 13.774
  07:54:33 STEP 2618 H=2618:00 +CPU= 14.498
  07:54:56 STEP 2619 H=2619:00 +CPU= 22.639
  07:55:10 STEP 2620 H=2620:00 +CPU= 14.256
  07:55:24 STEP 2621 H=2621:00 +CPU= 13.489
  07:55:47 STEP 2622 H=2622:00 +CPU= 22.959
  07:56:00 STEP 2623 H=2623:00 +CPU= 13.334
  07:56:14 STEP 2624 H=2624:00 +CPU= 13.786
  07:56:36 STEP 2625 H=2625:00 +CPU= 21.810
  07:56:49 STEP 2626 H=2626:00 +CPU= 13.597
  07:57:03 STEP 2627 H=2627:00 +CPU= 14.124
  07:57:28 STEP 2628 H=2628:00 +CPU= 24.332
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002628
  07:57:41 STEP 2629 H=2629:00 +CPU= 13.833
  07:57:55 STEP 2630 H=2630:00 +CPU= 14.085
  07:58:18 STEP 2631 H=2631:00 +CPU= 22.431
  07:58:31 STEP 2632 H=2632:00 +CPU= 13.536
Quit request received from BOINC client, ending the child process
..Failed, model did not complete successfully
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0077
(argv4) batchid: 958
(argv5) wuid: 12174721
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/17
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b34a0487c678faaa8e1b746ca63f788d to: /var/lib/boinc-client/slots/17/oifs_43r3_ps_0077_1989050100_123_958_12174721.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/17/oifs_43r3_ps_0077_1989050100_123_958_12174721.zip
ic_ancil_file: ic_ancil_12174721
ifsdata_file: ifsdata_12174721
climate_data_file: clim_data_12174721
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/17/ic_ancil_12174721.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12174721.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eb2c5f38983d7ba9fddbd7db3124a25a to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12174721.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12174721.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12174721.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12174721.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12174721.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12174721.xml
-- Model is restarting --
Adjusting last_iter, 2633, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12174721.xml
last_cpu_time: 46955
upload_file_number: 109
last_iter: 2615
last_upload: 9417600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124
The child process has been launched with process id: 163273
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:163273:163273] [20230116:013253:1673832773:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  01:35:11 STEP 2616 H=2616:00 +CPU=127.714
  01:36:28 STEP 2617 H=2617:00 +CPU= 76.885
  01:37:47 STEP 2618 H=2618:00 +CPU= 79.210
  01:39:31 STEP 2619 H=2619:00 +CPU=103.295
  01:40:45 STEP 2620 H=2620:00 +CPU= 74.120
  01:42:03 STEP 2621 H=2621:00 +CPU= 76.910
  01:43:44 STEP 2622 H=2622:00 +CPU=101.217
  01:44:59 STEP 2623 H=2623:00 +CPU= 74.779
  01:46:17 STEP 2624 H=2624:00 +CPU= 77.780
  01:47:54 STEP 2625 H=2625:00 +CPU= 96.397
  01:49:13 STEP 2626 H=2626:00 +CPU= 78.248
  01:50:29 STEP 2627 H=2627:00 +CPU= 75.951
  01:52:28 STEP 2628 H=2628:00 +CPU=118.831
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002628
  01:53:40 STEP 2629 H=2629:00 +CPU= 71.533
  01:54:59 STEP 2630 H=2630:00 +CPU= 78.967
  01:56:47 STEP 2631 H=2631:00 +CPU=107.921
  01:58:04 STEP 2632 H=2632:00 +CPU= 75.943
  01:59:17 STEP 2633 H=2633:00 +CPU= 72.823
  02:00:58 STEP 2634 H=2634:00 +CPU=101.202
  02:02:14 STEP 2635 H=2635:00 +CPU= 75.073
  02:03:27 STEP 2636 H=2636:00 +CPU= 72.779
  02:05:18 STEP 2637 H=2637:00 +CPU=110.134
  02:06:27 STEP 2638 H=2638:00 +CPU= 69.662
  02:07:42 STEP 2639 H=2639:00 +CPU= 74.822
  02:09:55 STEP 2640 H=2640:00 +CPU=132.040
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_109.zip
Uploading the intermediate file: upload_file_109.zip
  02:11:14 STEP 2641 H=2641:00 +CPU= 77.919
  02:12:23 STEP 2642 H=2642:00 +CPU= 68.367
  02:14:13 STEP 2643 H=2643:00 +CPU=109.704
  02:15:31 STEP 2644 H=2644:00 +CPU= 77.566
  02:16:48 STEP 2645 H=2645:00 +CPU= 75.765
  02:18:40 STEP 2646 H=2646:00 +CPU=110.734
  02:19:57 STEP 2647 H=2647:00 +CPU= 77.142
  02:21:14 STEP 2648 H=2648:00 +CPU= 76.708
  02:23:07 STEP 2649 H=2649:00 +CPU=112.164
  02:24:24 STEP 2650 H=2650:00 +CPU= 76.367
  02:25:41 STEP 2651 H=2651:00 +CPU= 76.487
  02:27:41 STEP 2652 H=2652:00 +CPU=119.646
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002652
  02:28:55 STEP 2653 H=2653:00 +CPU= 73.742
  02:30:05 STEP 2654 H=2654:00 +CPU= 69.748
  02:31:53 STEP 2655 H=2655:00 +CPU=107.735
  02:33:08 STEP 2656 H=2656:00 +CPU= 75.272
  02:34:23 STEP 2657 H=2657:00 +CPU= 74.287
  02:36:08 STEP 2658 H=2658:00 +CPU=104.437
  02:37:21 STEP 2659 H=2659:00 +CPU= 73.425
  02:38:40 STEP 2660 H=2660:00 +CPU= 77.999
  02:40:33 STEP 2661 H=2661:00 +CPU=112.945
  02:41:51 STEP 2662 H=2662:00 +CPU= 77.564
  02:43:12 STEP 2663 H=2663:00 +CPU= 80.756
  02:45:18 STEP 2664 H=2664:00 +CPU=125.256
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_110.zip
Uploading the intermediate file: upload_file_110.zip
  02:46:27 STEP 2665 H=2665:00 +CPU= 68.595
  02:47:46 STEP 2666 H=2666:00 +CPU= 78.579
  02:49:35 STEP 2667 H=2667:00 +CPU=108.684
  02:50:50 STEP 2668 H=2668:00 +CPU= 74.231
  02:52:04 STEP 2669 H=2669:00 +CPU= 74.108
  02:53:57 STEP 2670 H=2670:00 +CPU=112.658
  02:55:13 STEP 2671 H=2671:00 +CPU= 75.499
  02:56:31 STEP 2672 H=2672:00 +CPU= 77.593
  02:58:21 STEP 2673 H=2673:00 +CPU=108.988
  02:59:40 STEP 2674 H=2674:00 +CPU= 78.580
  03:00:55 STEP 2675 H=2675:00 +CPU= 74.984
  03:02:58 STEP 2676 H=2676:00 +CPU=122.679
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002676
  03:04:12 STEP 2677 H=2677:00 +CPU= 73.003
  03:05:32 STEP 2678 H=2678:00 +CPU= 80.226
  03:07:15 STEP 2679 H=2679:00 +CPU=102.588
  03:08:25 STEP 2680 H=2680:00 +CPU= 69.714
  03:09:40 STEP 2681 H=2681:00 +CPU= 75.038
  03:11:28 STEP 2682 H=2682:00 +CPU=107.374
  03:12:43 STEP 2683 H=2683:00 +CPU= 74.817
  03:13:59 STEP 2684 H=2684:00 +CPU= 75.693
  03:15:48 STEP 2685 H=2685:00 +CPU=108.504
  03:17:05 STEP 2686 H=2686:00 +CPU= 76.180
  03:18:23 STEP 2687 H=2687:00 +CPU= 77.472
  03:20:31 STEP 2688 H=2688:00 +CPU=127.594
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_111.zip
Uploading the intermediate file: upload_file_111.zip
  03:21:46 STEP 2689 H=2689:00 +CPU= 74.733
  03:23:05 STEP 2690 H=2690:00 +CPU= 77.907
  03:24:58 STEP 2691 H=2691:00 +CPU=112.151
  03:26:17 STEP 2692 H=2692:00 +CPU= 78.403
  03:27:35 STEP 2693 H=2693:00 +CPU= 77.855
  03:29:23 STEP 2694 H=2694:00 +CPU=107.295
  03:30:39 STEP 2695 H=2695:00 +CPU= 75.628
  03:31:54 STEP 2696 H=2696:00 +CPU= 75.399
  03:33:47 STEP 2697 H=2697:00 +CPU=112.169
  03:35:09 STEP 2698 H=2698:00 +CPU= 81.893
  03:36:28 STEP 2699 H=2699:00 +CPU= 78.631
  03:38:32 STEP 2700 H=2700:00 +CPU=123.386
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002700
  03:39:40 STEP 2701 H=2701:00 +CPU= 68.357
  03:40:59 STEP 2702 H=2702:00 +CPU= 78.111
  03:42:41 STEP 2703 H=2703:00 +CPU=102.535
  03:43:55 STEP 2704 H=2704:00 +CPU= 73.180
  03:45:14 STEP 2705 H=2705:00 +CPU= 79.053
  03:47:08 STEP 2706 H=2706:00 +CPU=113.109
  03:48:25 STEP 2707 H=2707:00 +CPU= 76.803
  03:49:45 STEP 2708 H=2708:00 +CPU= 79.828
  03:51:33 STEP 2709 H=2709:00 +CPU=107.718
  03:52:47 STEP 2710 H=2710:00 +CPU= 72.694
  03:54:06 STEP 2711 H=2711:00 +CPU= 78.594
  03:56:18 STEP 2712 H=2712:00 +CPU=130.669
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_112.zip
Uploading the intermediate file: upload_file_112.zip
  03:57:33 STEP 2713 H=2713:00 +CPU= 74.877
  03:58:48 STEP 2714 H=2714:00 +CPU= 74.825
  04:00:39 STEP 2715 H=2715:00 +CPU=110.128
  04:01:58 STEP 2716 H=2716:00 +CPU= 78.526
  04:03:17 STEP 2717 H=2717:00 +CPU= 77.110
  04:05:09 STEP 2718 H=2718:00 +CPU=111.323
  04:06:24 STEP 2719 H=2719:00 +CPU= 74.619
  04:07:43 STEP 2720 H=2720:00 +CPU= 78.860
  04:09:40 STEP 2721 H=2721:00 +CPU=115.769
  04:10:59 STEP 2722 H=2722:00 +CPU= 79.473
  04:12:16 STEP 2723 H=2723:00 +CPU= 76.219
  04:14:15 STEP 2724 H=2724:00 +CPU=118.550
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002724
  04:15:29 STEP 2725 H=2725:00 +CPU= 73.261
  04:16:47 STEP 2726 H=2726:00 +CPU= 78.426
  04:18:37 STEP 2727 H=2727:00 +CPU=109.647
  04:19:57 STEP 2728 H=2728:00 +CPU= 79.179
  04:21:12 STEP 2729 H=2729:00 +CPU= 74.244
  04:23:05 STEP 2730 H=2730:00 +CPU=112.081
  04:24:17 STEP 2731 H=2731:00 +CPU= 72.070
  04:25:36 STEP 2732 H=2732:00 +CPU= 79.251
  04:27:14 STEP 2733 H=2733:00 +CPU= 97.502
  04:28:31 STEP 2734 H=2734:00 +CPU= 76.858
  04:29:49 STEP 2735 H=2735:00 +CPU= 77.534
  04:31:54 STEP 2736 H=2736:00 +CPU=123.919
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_113.zip
Uploading the intermediate file: upload_file_113.zip
  04:33:09 STEP 2737 H=2737:00 +CPU= 75.061
  04:34:26 STEP 2738 H=2738:00 +CPU= 75.739
  04:36:17 STEP 2739 H=2739:00 +CPU=110.584
  04:37:35 STEP 2740 H=2740:00 +CPU= 78.304
  04:38:51 STEP 2741 H=2741:00 +CPU= 75.693
  04:40:47 STEP 2742 H=2742:00 +CPU=114.927
  04:42:06 STEP 2743 H=2743:00 +CPU= 77.910
  04:43:19 STEP 2744 H=2744:00 +CPU= 73.747
  04:45:11 STEP 2745 H=2745:00 +CPU=111.126
  04:46:25 STEP 2746 H=2746:00 +CPU= 73.028
  04:47:38 STEP 2747 H=2747:00 +CPU= 73.369
  04:49:38 STEP 2748 H=2748:00 +CPU=119.886
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002748
  04:50:53 STEP 2749 H=2749:00 +CPU= 74.678
  04:52:17 STEP 2750 H=2750:00 +CPU= 83.338
  04:54:06 STEP 2751 H=2751:00 +CPU=108.024
  04:55:21 STEP 2752 H=2752:00 +CPU= 74.951
  04:56:36 STEP 2753 H=2753:00 +CPU= 75.058
  04:58:35 STEP 2754 H=2754:00 +CPU=118.111
  04:59:50 STEP 2755 H=2755:00 +CPU= 75.666
  05:01:07 STEP 2756 H=2756:00 +CPU= 76.508
  05:03:04 STEP 2757 H=2757:00 +CPU=116.814
  05:04:17 STEP 2758 H=2758:00 +CPU= 71.832
  05:05:35 STEP 2759 H=2759:00 +CPU= 77.501
  05:07:44 STEP 2760 H=2760:00 +CPU=127.953
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_114.zip
Uploading the intermediate file: upload_file_114.zip
  05:09:03 STEP 2761 H=2761:00 +CPU= 78.477
  05:10:16 STEP 2762 H=2762:00 +CPU= 73.024
  05:12:05 STEP 2763 H=2763:00 +CPU=108.647
  05:13:24 STEP 2764 H=2764:00 +CPU= 78.726
  05:14:42 STEP 2765 H=2765:00 +CPU= 77.166
  05:16:32 STEP 2766 H=2766:00 +CPU=110.573
  05:17:42 STEP 2767 H=2767:00 +CPU= 69.491
  05:19:01 STEP 2768 H=2768:00 +CPU= 78.984
  05:20:36 STEP 2769 H=2769:00 +CPU= 94.782
  05:21:53 STEP 2770 H=2770:00 +CPU= 76.150
  05:23:09 STEP 2771 H=2771:00 +CPU= 75.928
  05:25:06 STEP 2772 H=2772:00 +CPU=116.303
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002772
  05:26:21 STEP 2773 H=2773:00 +CPU= 75.215
  05:27:35 STEP 2774 H=2774:00 +CPU= 73.652
  05:29:26 STEP 2775 H=2775:00 +CPU=110.359
  05:30:46 STEP 2776 H=2776:00 +CPU= 79.416
  05:31:59 STEP 2777 H=2777:00 +CPU= 73.628
  05:33:55 STEP 2778 H=2778:00 +CPU=115.246
  05:35:07 STEP 2779 H=2779:00 +CPU= 72.063
  05:36:23 STEP 2780 H=2780:00 +CPU= 75.668
  05:38:04 STEP 2781 H=2781:00 +CPU=100.157
  05:39:24 STEP 2782 H=2782:00 +CPU= 79.904
  05:40:43 STEP 2783 H=2783:00 +CPU= 78.874
  05:42:55 STEP 2784 H=2784:00 +CPU=131.383
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_115.zip
Uploading the intermediate file: upload_file_115.zip
  05:44:13 STEP 2785 H=2785:00 +CPU= 77.009
  05:45:31 STEP 2786 H=2786:00 +CPU= 77.523
  05:47:19 STEP 2787 H=2787:00 +CPU=108.414
  05:48:38 STEP 2788 H=2788:00 +CPU= 77.904
  05:49:57 STEP 2789 H=2789:00 +CPU= 78.835
  05:51:43 STEP 2790 H=2790:00 +CPU=106.215
  05:53:01 STEP 2791 H=2791:00 +CPU= 77.837
  05:54:22 STEP 2792 H=2792:00 +CPU= 80.143
  05:56:13 STEP 2793 H=2793:00 +CPU=110.769
  05:57:29 STEP 2794 H=2794:00 +CPU= 75.694
  05:58:44 STEP 2795 H=2795:00 +CPU= 74.859
  06:00:49 STEP 2796 H=2796:00 +CPU=124.142
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002796
  06:02:06 STEP 2797 H=2797:00 +CPU= 76.607
  06:03:20 STEP 2798 H=2798:00 +CPU= 73.759
  06:05:12 STEP 2799 H=2799:00 +CPU=110.993
  06:06:28 STEP 2800 H=2800:00 +CPU= 75.216
  06:07:48 STEP 2801 H=2801:00 +CPU= 80.556
  06:09:41 STEP 2802 H=2802:00 +CPU=112.765
  06:10:58 STEP 2803 H=2803:00 +CPU= 75.703
  06:12:15 STEP 2804 H=2804:00 +CPU= 76.834
  06:14:06 STEP 2805 H=2805:00 +CPU=110.714
  06:15:23 STEP 2806 H=2806:00 +CPU= 76.250
  06:16:37 STEP 2807 H=2807:00 +CPU= 73.854
  06:18:53 STEP 2808 H=2808:00 +CPU=136.037
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_116.zip
Uploading the intermediate file: upload_file_116.zip
  06:20:10 STEP 2809 H=2809:00 +CPU= 76.792
  06:21:26 STEP 2810 H=2810:00 +CPU= 75.579
  06:23:14 STEP 2811 H=2811:00 +CPU=107.521
  06:24:32 STEP 2812 H=2812:00 +CPU= 77.313
  06:25:47 STEP 2813 H=2813:00 +CPU= 75.036
  06:27:40 STEP 2814 H=2814:00 +CPU=111.812
  06:28:57 STEP 2815 H=2815:00 +CPU= 76.784
  06:30:15 STEP 2816 H=2816:00 +CPU= 77.305
  06:32:04 STEP 2817 H=2817:00 +CPU=108.655
  06:33:17 STEP 2818 H=2818:00 +CPU= 72.069
  06:34:30 STEP 2819 H=2819:00 +CPU= 72.783
  06:36:34 STEP 2820 H=2820:00 +CPU=123.069
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002820
  06:37:48 STEP 2821 H=2821:00 +CPU= 73.864
  06:39:05 STEP 2822 H=2822:00 +CPU= 76.450
  06:40:45 STEP 2823 H=2823:00 +CPU= 99.958
  06:41:57 STEP 2824 H=2824:00 +CPU= 72.099
  06:43:15 STEP 2825 H=2825:00 +CPU= 77.595
  06:45:10 STEP 2826 H=2826:00 +CPU=114.404
  06:46:22 STEP 2827 H=2827:00 +CPU= 71.711
  06:47:38 STEP 2828 H=2828:00 +CPU= 75.764
  06:49:24 STEP 2829 H=2829:00 +CPU=105.807
  06:50:37 STEP 2830 H=2830:00 +CPU= 72.850
  06:51:57 STEP 2831 H=2831:00 +CPU= 79.245
  06:54:01 STEP 2832 H=2832:00 +CPU=123.339
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_117.zip
Uploading the intermediate file: upload_file_117.zip
  06:55:19 STEP 2833 H=2833:00 +CPU= 77.722
  06:56:35 STEP 2834 H=2834:00 +CPU= 75.782
  06:58:25 STEP 2835 H=2835:00 +CPU=108.957
  06:59:43 STEP 2836 H=2836:00 +CPU= 77.184
  07:01:01 STEP 2837 H=2837:00 +CPU= 77.667
  07:02:54 STEP 2838 H=2838:00 +CPU=112.426
  07:04:10 STEP 2839 H=2839:00 +CPU= 75.125
  07:05:26 STEP 2840 H=2840:00 +CPU= 75.467
  07:07:13 STEP 2841 H=2841:00 +CPU=105.410
  07:08:28 STEP 2842 H=2842:00 +CPU= 75.202
  07:09:34 STEP 2843 H=2843:00 +CPU= 65.741
  07:11:33 STEP 2844 H=2844:00 +CPU=118.535
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002844
  07:12:48 STEP 2845 H=2845:00 +CPU= 73.858
  07:14:04 STEP 2846 H=2846:00 +CPU= 75.411
  07:15:54 STEP 2847 H=2847:00 +CPU=110.076
  07:17:07 STEP 2848 H=2848:00 +CPU= 71.975
  07:18:25 STEP 2849 H=2849:00 +CPU= 78.237
  07:20:19 STEP 2850 H=2850:00 +CPU=113.161
  07:21:35 STEP 2851 H=2851:00 +CPU= 75.556
  07:22:51 STEP 2852 H=2852:00 +CPU= 75.416
  07:24:43 STEP 2853 H=2853:00 +CPU=111.942
  07:25:55 STEP 2854 H=2854:00 +CPU= 71.958
  07:27:13 STEP 2855 H=2855:00 +CPU= 77.819
  07:29:25 STEP 2856 H=2856:00 +CPU=130.790
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_118.zip
Uploading the intermediate file: upload_file_118.zip
Uploading trickle at timestep: 10281600
  07:30:37 STEP 2857 H=2857:00 +CPU= 71.865
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1989050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0077
(argv4) batchid: 958
(argv5) wuid: 12174721
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/17
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/17/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_b34a0487c678faaa8e1b746ca63f788d to: /var/lib/boinc-client/slots/17/oifs_43r3_ps_0077_1989050100_123_958_12174721.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/17/oifs_43r3_ps_0077_1989050100_123_958_12174721.zip
ic_ancil_file: ic_ancil_12174721
ifsdata_file: ifsdata_12174721
climate_data_file: clim_data_12174721
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d0c0d3d8314ac7e69746fb25c162455c to: /var/lib/boinc-client/slots/17/ic_ancil_12174721.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/17/ic_ancil_12174721.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eb2c5f38983d7ba9fddbd7db3124a25a to: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12174721.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/17/ifsdata/ifsdata_12174721.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/17/159l_2/clim_data_12174721.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/17/159l_2/clim_data_12174721.zip
Checking for progress XML file: /var/lib/boinc-client/slots/17/progress_file_12174721.xml
Opened progress file ok : /var/lib/boinc-client/slots/17/progress_file_12174721.xml
-- Model is restarting --
Adjusting last_iter, 2858, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/17/progress_file_12174721.xml
last_cpu_time: 68359
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124
The child process has been launched with process id: 179428
Executing the command: /var/lib/boinc-client/slots/17/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:PREMIUM2:1:1:179428:179428] [20230116:204726:1673902046:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:49:48 STEP 2856 H=2856:00 +CPU=131.192
  20:51:06 STEP 2857 H=2857:00 +CPU= 77.857
  20:52:25 STEP 2858 H=2858:00 +CPU= 78.969
  20:54:16 STEP 2859 H=2859:00 +CPU=110.482
  20:55:35 STEP 2860 H=2860:00 +CPU= 78.803
  20:56:51 STEP 2861 H=2861:00 +CPU= 75.803
  20:58:42 STEP 2862 H=2862:00 +CPU=110.467
  20:59:59 STEP 2863 H=2863:00 +CPU= 76.617
  21:01:15 STEP 2864 H=2864:00 +CPU= 75.636
  21:03:04 STEP 2865 H=2865:00 +CPU=109.195
  21:04:23 STEP 2866 H=2866:00 +CPU= 78.676
  21:05:43 STEP 2867 H=2867:00 +CPU= 79.538
  21:07:36 STEP 2868 H=2868:00 +CPU=112.375
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002868
  21:08:49 STEP 2869 H=2869:00 +CPU= 72.716
  21:10:01 STEP 2870 H=2870:00 +CPU= 72.482
  21:11:48 STEP 2871 H=2871:00 +CPU=106.022
  21:12:59 STEP 2872 H=2872:00 +CPU= 70.904
  21:14:11 STEP 2873 H=2873:00 +CPU= 71.167
  21:16:00 STEP 2874 H=2874:00 +CPU=108.149
  21:17:19 STEP 2875 H=2875:00 +CPU= 78.085
  21:18:37 STEP 2876 H=2876:00 +CPU= 77.387
  21:20:27 STEP 2877 H=2877:00 +CPU=109.587
  21:21:41 STEP 2878 H=2878:00 +CPU= 73.772
  21:22:59 STEP 2879 H=2879:00 +CPU= 77.307
  21:25:12 STEP 2880 H=2880:00 +CPU=131.894
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_119.zip
Uploading the intermediate file: upload_file_119.zip
  21:26:30 STEP 2881 H=2881:00 +CPU= 77.453
  21:27:48 STEP 2882 H=2882:00 +CPU= 78.087
  21:29:39 STEP 2883 H=2883:00 +CPU=110.463
  21:31:00 STEP 2884 H=2884:00 +CPU= 80.806
  21:32:16 STEP 2885 H=2885:00 +CPU= 75.705
  21:34:07 STEP 2886 H=2886:00 +CPU=110.278
  21:35:24 STEP 2887 H=2887:00 +CPU= 76.635
  21:36:41 STEP 2888 H=2888:00 +CPU= 76.504
  21:38:29 STEP 2889 H=2889:00 +CPU=107.144
  21:39:49 STEP 2890 H=2890:00 +CPU= 79.667
  21:41:02 STEP 2891 H=2891:00 +CPU= 72.754
  21:43:01 STEP 2892 H=2892:00 +CPU=119.122
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002892
  21:44:17 STEP 2893 H=2893:00 +CPU= 75.908
  21:45:35 STEP 2894 H=2894:00 +CPU= 77.155
  21:47:28 STEP 2895 H=2895:00 +CPU=113.331
  21:48:51 STEP 2896 H=2896:00 +CPU= 82.044
  21:50:07 STEP 2897 H=2897:00 +CPU= 75.187
  21:52:03 STEP 2898 H=2898:00 +CPU=116.020
  21:53:15 STEP 2899 H=2899:00 +CPU= 71.287
  21:54:31 STEP 2900 H=2900:00 +CPU= 76.564
  21:56:19 STEP 2901 H=2901:00 +CPU=107.585
  21:57:38 STEP 2902 H=2902:00 +CPU= 78.904
  21:58:59 STEP 2903 H=2903:00 +CPU= 79.910
  22:01:12 STEP 2904 H=2904:00 +CPU=132.789
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_120.zip
Uploading the intermediate file: upload_file_120.zip
  22:02:31 STEP 2905 H=2905:00 +CPU= 78.607
  22:03:47 STEP 2906 H=2906:00 +CPU= 74.703
  22:05:37 STEP 2907 H=2907:00 +CPU=109.561
  22:06:58 STEP 2908 H=2908:00 +CPU= 81.093
  22:08:15 STEP 2909 H=2909:00 +CPU= 76.592
  22:10:08 STEP 2910 H=2910:00 +CPU=112.718
  22:11:21 STEP 2911 H=2911:00 +CPU= 72.849
  22:12:41 STEP 2912 H=2912:00 +CPU= 80.194
  22:14:33 STEP 2913 H=2913:00 +CPU=111.597
  22:15:53 STEP 2914 H=2914:00 +CPU= 79.296
  22:17:06 STEP 2915 H=2915:00 +CPU= 72.753
  22:19:09 STEP 2916 H=2916:00 +CPU=121.572
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002916
  22:20:22 STEP 2917 H=2917:00 +CPU= 73.130
  22:21:36 STEP 2918 H=2918:00 +CPU= 73.021
  22:23:24 STEP 2919 H=2919:00 +CPU=107.585
  22:24:41 STEP 2920 H=2920:00 +CPU= 76.635
  22:25:54 STEP 2921 H=2921:00 +CPU= 72.885
  22:27:45 STEP 2922 H=2922:00 +CPU=111.167
  22:29:01 STEP 2923 H=2923:00 +CPU= 74.803
  22:30:14 STEP 2924 H=2924:00 +CPU= 72.487
  22:32:03 STEP 2925 H=2925:00 +CPU=108.894
  22:33:19 STEP 2926 H=2926:00 +CPU= 75.726
  22:34:37 STEP 2927 H=2927:00 +CPU= 77.866
  22:36:29 STEP 2928 H=2928:00 +CPU=111.213
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_121.zip
Uploading the intermediate file: upload_file_121.zip
  22:37:38 STEP 2929 H=2929:00 +CPU= 68.623
  22:38:51 STEP 2930 H=2930:00 +CPU= 73.261
  22:40:37 STEP 2931 H=2931:00 +CPU=104.975
  22:41:53 STEP 2932 H=2932:00 +CPU= 75.625
  22:43:08 STEP 2933 H=2933:00 +CPU= 75.103
  22:44:55 STEP 2934 H=2934:00 +CPU=106.003
  22:46:11 STEP 2935 H=2935:00 +CPU= 76.370
  22:47:26 STEP 2936 H=2936:00 +CPU= 74.516
  22:49:13 STEP 2937 H=2937:00 +CPU=106.117
  22:50:29 STEP 2938 H=2938:00 +CPU= 75.291
  22:51:42 STEP 2939 H=2939:00 +CPU= 72.724
  22:53:36 STEP 2940 H=2940:00 +CPU=113.631
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002940
  22:54:53 STEP 2941 H=2941:00 +CPU= 76.594
  22:56:07 STEP 2942 H=2942:00 +CPU= 73.912
  22:57:57 STEP 2943 H=2943:00 +CPU=108.507
  22:59:14 STEP 2944 H=2944:00 +CPU= 77.139
  23:00:30 STEP 2945 H=2945:00 +CPU= 75.107
  23:02:15 STEP 2946 H=2946:00 +CPU=104.368
  23:03:32 STEP 2947 H=2947:00 +CPU= 77.407
  23:04:44 STEP 2948 H=2948:00 +CPU= 71.738
  23:06:39 STEP 2949 H=2949:00 +CPU=113.553
  23:07:55 STEP 2950 H=2950:00 +CPU= 75.788
  23:09:16 STEP 2951 H=2951:00 +CPU= 79.861
  23:11:21 STEP 2952 H=2952:00 +CPU=124.150
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc-client/slots/17/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMSHhq0f+002940
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12174721/ICMGGhq0f+002928
Zipping up the final file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0077_1989050100_123_958_12174721_2_r1196740124_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
23:14:45 (179425): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:23:10 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 10,623,600 76,952 0.0072
09 May 2023 16:22:02 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 10,281,600 68,247 0.0066
09 May 2023 16:20:21 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 8,640,000 42,922 0.0050
09 May 2023 16:20:17 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 7,776,000 38,679 0.0050
09 May 2023 16:20:13 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 6,912,000 34,485 0.0050
09 May 2023 16:20:09 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 6,048,000 30,303 0.0050
09 May 2023 16:20:05 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 5,184,000 26,130 0.0050
09 May 2023 16:19:57 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 4,320,000 21,788 0.0050
09 May 2023 16:19:53 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 3,456,000 17,255 0.0050
09 May 2023 16:19:49 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 2,592,000 12,763 0.0049
09 May 2023 16:19:45 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 1,728,000 8,411 0.0049
09 May 2023 16:19:42 1532338 22297906 oifs_43r3_ps_0077_1989050100_123_958_12174721_2 864,000 4,189 0.0048


©2024 climateprediction.net