climateprediction.net home page
Task 22317004

Task 22317004

Name oifs_43r3_0189_2019110100_123_993_12215031_1
Workunit 12215031
Created 25 Feb 2023, 6:26:13 UTC
Sent 25 Feb 2023, 6:26:35 UTC
Report deadline 26 Apr 2023, 6:26:35 UTC
Received 3 Mar 2023, 14:41:37 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1467458
Run time 17 hours 38 min 16 sec
CPU time 17 hours 33 min 37 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 5.99 GFLOPS
Application version OpenIFS 43r3 v1.21
x86_64-pc-linux-gnu
Peak working set size 4,425.53 MB
Peak swap size 4,984.14 MB
Peak disk usage 2,008.13 MB
Stderr
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<stderr_txt>
_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:168747:168747] [20230303:111631:1677842191:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  11:17:01 STEP 2496 H=2496:00 +CPU= 27.565
  11:17:17 STEP 2497 H=2497:00 +CPU= 15.953
  11:17:34 STEP 2498 H=2498:00 +CPU= 16.085
  11:17:58 STEP 2499 H=2499:00 +CPU= 24.113
  11:18:15 STEP 2500 H=2500:00 +CPU= 16.492
  11:18:31 STEP 2501 H=2501:00 +CPU= 15.885
  11:18:57 STEP 2502 H=2502:00 +CPU= 25.367
  11:19:12 STEP 2503 H=2503:00 +CPU= 15.309
  11:19:29 STEP 2504 H=2504:00 +CPU= 16.192
  11:19:53 STEP 2505 H=2505:00 +CPU= 24.117
  11:20:09 STEP 2506 H=2506:00 +CPU= 16.099
  11:20:25 STEP 2507 H=2507:00 +CPU= 15.639
  11:20:52 STEP 2508 H=2508:00 +CPU= 26.869
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002508
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002508
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002508
  11:21:08 STEP 2509 H=2509:00 +CPU= 15.304
  11:21:24 STEP 2510 H=2510:00 +CPU= 16.015
  11:21:48 STEP 2511 H=2511:00 +CPU= 23.989
  11:22:04 STEP 2512 H=2512:00 +CPU= 15.938
  11:22:20 STEP 2513 H=2513:00 +CPU= 15.346
  11:22:45 STEP 2514 H=2514:00 +CPU= 25.381
  11:23:01 STEP 2515 H=2515:00 +CPU= 15.214
  11:23:17 STEP 2516 H=2516:00 +CPU= 15.669
  11:23:41 STEP 2517 H=2517:00 +CPU= 24.630
  11:23:58 STEP 2518 H=2518:00 +CPU= 16.088
  11:24:14 STEP 2519 H=2519:00 +CPU= 16.186
  11:24:43 STEP 2520 H=2520:00 +CPU= 28.547
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002520
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002520
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002520
  11:24:59 STEP 2521 H=2521:00 +CPU= 16.306
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002520
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_104.zip
Uploading the intermediate file: upload_file_104.zip
  11:25:16 STEP 2522 H=2522:00 +CPU= 16.009
  11:25:40 STEP 2523 H=2523:00 +CPU= 24.258
  11:25:57 STEP 2524 H=2524:00 +CPU= 16.285
  11:26:13 STEP 2525 H=2525:00 +CPU= 16.293
  11:26:39 STEP 2526 H=2526:00 +CPU= 25.359
  11:26:54 STEP 2527 H=2527:00 +CPU= 15.486
  11:27:11 STEP 2528 H=2528:00 +CPU= 16.100
  11:27:35 STEP 2529 H=2529:00 +CPU= 24.384
  11:27:51 STEP 2530 H=2530:00 +CPU= 16.046
  11:28:08 STEP 2531 H=2531:00 +CPU= 16.038
  11:28:34 STEP 2532 H=2532:00 +CPU= 26.363
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002532
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002532
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002532
  11:28:51 STEP 2533 H=2533:00 +CPU= 15.883
  11:29:07 STEP 2534 H=2534:00 +CPU= 16.026
  11:29:31 STEP 2535 H=2535:00 +CPU= 24.314
  11:29:48 STEP 2536 H=2536:00 +CPU= 16.259
  11:30:04 STEP 2537 H=2537:00 +CPU= 15.642
  11:30:28 STEP 2538 H=2538:00 +CPU= 24.496
  11:30:44 STEP 2539 H=2539:00 +CPU= 15.673
  11:31:01 STEP 2540 H=2540:00 +CPU= 16.137
  11:31:25 STEP 2541 H=2541:00 +CPU= 24.234
  11:31:41 STEP 2542 H=2542:00 +CPU= 15.653
  11:31:57 STEP 2543 H=2543:00 +CPU= 16.091
  11:32:26 STEP 2544 H=2544:00 +CPU= 28.547
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002544
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002544
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002544
  11:32:42 STEP 2545 H=2545:00 +CPU= 15.669
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002544
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_105.zip
Uploading the intermediate file: upload_file_105.zip
  11:32:58 STEP 2546 H=2546:00 +CPU= 16.039
  11:33:23 STEP 2547 H=2547:00 +CPU= 24.249
  11:33:39 STEP 2548 H=2548:00 +CPU= 16.074
  11:33:55 STEP 2549 H=2549:00 +CPU= 15.370
  11:34:19 STEP 2550 H=2550:00 +CPU= 24.531
  11:34:35 STEP 2551 H=2551:00 +CPU= 15.572
  11:34:52 STEP 2552 H=2552:00 +CPU= 16.319
  11:35:17 STEP 2553 H=2553:00 +CPU= 24.397
  11:35:33 STEP 2554 H=2554:00 +CPU= 15.984
  11:35:48 STEP 2555 H=2555:00 +CPU= 15.436
  11:36:15 STEP 2556 H=2556:00 +CPU= 26.760
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002556
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002556
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002556
  11:36:31 STEP 2557 H=2557:00 +CPU= 15.348
  11:36:47 STEP 2558 H=2558:00 +CPU= 15.735
  11:37:12 STEP 2559 H=2559:00 +CPU= 24.584
  11:37:27 STEP 2560 H=2560:00 +CPU= 15.426
  11:37:43 STEP 2561 H=2561:00 +CPU= 15.950
  11:38:08 STEP 2562 H=2562:00 +CPU= 24.737
  11:38:24 STEP 2563 H=2563:00 +CPU= 15.568
  11:38:40 STEP 2564 H=2564:00 +CPU= 15.935
  11:39:05 STEP 2565 H=2565:00 +CPU= 24.618
  11:39:21 STEP 2566 H=2566:00 +CPU= 15.945
  11:39:38 STEP 2567 H=2567:00 +CPU= 16.035
  11:40:06 STEP 2568 H=2568:00 +CPU= 28.447
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002568
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002568
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002568
  11:40:22 STEP 2569 H=2569:00 +CPU= 15.812
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002568
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_106.zip
Uploading the intermediate file: upload_file_106.zip
  11:40:39 STEP 2570 H=2570:00 +CPU= 16.166
  11:41:03 STEP 2571 H=2571:00 +CPU= 24.300
  11:41:20 STEP 2572 H=2572:00 +CPU= 16.269
  11:41:36 STEP 2573 H=2573:00 +CPU= 16.136
  11:42:02 STEP 2574 H=2574:00 +CPU= 25.128
  11:42:17 STEP 2575 H=2575:00 +CPU= 15.639
  11:42:34 STEP 2576 H=2576:00 +CPU= 16.781
  11:42:59 STEP 2577 H=2577:00 +CPU= 24.709
  11:43:16 STEP 2578 H=2578:00 +CPU= 16.026
  11:43:31 STEP 2579 H=2579:00 +CPU= 15.534
  11:43:58 STEP 2580 H=2580:00 +CPU= 26.475
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002580
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002580
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002580
  11:44:14 STEP 2581 H=2581:00 +CPU= 15.674
  11:44:30 STEP 2582 H=2582:00 +CPU= 15.562
  11:44:54 STEP 2583 H=2583:00 +CPU= 24.317
  11:45:11 STEP 2584 H=2584:00 +CPU= 16.197
  11:45:26 STEP 2585 H=2585:00 +CPU= 15.507
  11:45:51 STEP 2586 H=2586:00 +CPU= 24.492
  11:46:07 STEP 2587 H=2587:00 +CPU= 15.318
  11:46:23 STEP 2588 H=2588:00 +CPU= 16.358
  11:46:48 STEP 2589 H=2589:00 +CPU= 24.117
  11:47:04 STEP 2590 H=2590:00 +CPU= 15.994
  11:47:20 STEP 2591 H=2591:00 +CPU= 15.797
  11:47:49 STEP 2592 H=2592:00 +CPU= 28.846
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002592
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002592
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002592
  11:48:04 STEP 2593 H=2593:00 +CPU= 15.263
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002592
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_107.zip
Uploading the intermediate file: upload_file_107.zip
  11:48:21 STEP 2594 H=2594:00 +CPU= 16.096
  11:48:45 STEP 2595 H=2595:00 +CPU= 23.684
  11:49:01 STEP 2596 H=2596:00 +CPU= 16.281
  11:49:17 STEP 2597 H=2597:00 +CPU= 15.206
  11:49:42 STEP 2598 H=2598:00 +CPU= 24.727
  11:49:57 STEP 2599 H=2599:00 +CPU= 15.006
  11:50:13 STEP 2600 H=2600:00 +CPU= 16.046
  11:50:37 STEP 2601 H=2601:00 +CPU= 24.233
  11:50:54 STEP 2602 H=2602:00 +CPU= 16.368
  11:51:10 STEP 2603 H=2603:00 +CPU= 15.557
  11:51:37 STEP 2604 H=2604:00 +CPU= 26.842
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002604
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002604
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002604
  11:51:52 STEP 2605 H=2605:00 +CPU= 15.406
  11:52:09 STEP 2606 H=2606:00 +CPU= 16.428
  11:52:32 STEP 2607 H=2607:00 +CPU= 23.258
  11:52:48 STEP 2608 H=2608:00 +CPU= 15.509
  11:53:04 STEP 2609 H=2609:00 +CPU= 15.475
  11:53:29 STEP 2610 H=2610:00 +CPU= 25.054
  11:53:45 STEP 2611 H=2611:00 +CPU= 15.435
  11:54:01 STEP 2612 H=2612:00 +CPU= 15.823
  11:54:26 STEP 2613 H=2613:00 +CPU= 24.563
  11:54:42 STEP 2614 H=2614:00 +CPU= 16.157
  11:54:58 STEP 2615 H=2615:00 +CPU= 16.173
  11:55:27 STEP 2616 H=2616:00 +CPU= 28.329
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002616
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002616
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002616
  11:55:42 STEP 2617 H=2617:00 +CPU= 15.263
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002616
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_108.zip
Uploading the intermediate file: upload_file_108.zip
  11:55:59 STEP 2618 H=2618:00 +CPU= 16.137
  11:56:24 STEP 2619 H=2619:00 +CPU= 24.637
  11:56:39 STEP 2620 H=2620:00 +CPU= 15.207
  11:56:55 STEP 2621 H=2621:00 +CPU= 15.521
  11:57:20 STEP 2622 H=2622:00 +CPU= 24.920
  11:57:36 STEP 2623 H=2623:00 +CPU= 16.132
  11:57:52 STEP 2624 H=2624:00 +CPU= 15.712
  11:58:16 STEP 2625 H=2625:00 +CPU= 24.265
  11:58:33 STEP 2626 H=2626:00 +CPU= 16.269
  11:58:48 STEP 2627 H=2627:00 +CPU= 15.385
  11:59:15 STEP 2628 H=2628:00 +CPU= 26.293
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002628
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002628
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002628
  11:59:31 STEP 2629 H=2629:00 +CPU= 15.793
  11:59:47 STEP 2630 H=2630:00 +CPU= 15.611
  12:00:11 STEP 2631 H=2631:00 +CPU= 24.323
  12:00:28 STEP 2632 H=2632:00 +CPU= 16.111
  12:00:43 STEP 2633 H=2633:00 +CPU= 15.435
  12:01:08 STEP 2634 H=2634:00 +CPU= 24.405
  12:01:24 STEP 2635 H=2635:00 +CPU= 15.598
  12:01:40 STEP 2636 H=2636:00 +CPU= 15.683
  12:02:04 STEP 2637 H=2637:00 +CPU= 24.014
  12:02:20 STEP 2638 H=2638:00 +CPU= 15.910
  12:02:36 STEP 2639 H=2639:00 +CPU= 15.946
  12:03:05 STEP 2640 H=2640:00 +CPU= 28.448
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002640
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002640
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002640
  12:03:21 STEP 2641 H=2641:00 +CPU= 15.580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002640
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_109.zip
Uploading the intermediate file: upload_file_109.zip
  12:03:37 STEP 2642 H=2642:00 +CPU= 16.260
  12:04:02 STEP 2643 H=2643:00 +CPU= 24.281
  12:04:18 STEP 2644 H=2644:00 +CPU= 16.364
  12:04:34 STEP 2645 H=2645:00 +CPU= 15.379
  12:04:59 STEP 2646 H=2646:00 +CPU= 24.844
  12:05:14 STEP 2647 H=2647:00 +CPU= 15.362
  12:05:30 STEP 2648 H=2648:00 +CPU= 15.794
  12:05:55 STEP 2649 H=2649:00 +CPU= 24.253
  12:06:11 STEP 2650 H=2650:00 +CPU= 16.063
  12:06:27 STEP 2651 H=2651:00 +CPU= 15.249
  12:06:54 STEP 2652 H=2652:00 +CPU= 26.628
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002652
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002652
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002652
  12:07:09 STEP 2653 H=2653:00 +CPU= 15.392
  12:07:25 STEP 2654 H=2654:00 +CPU= 15.989
  12:07:49 STEP 2655 H=2655:00 +CPU= 23.731
  12:08:06 STEP 2656 H=2656:00 +CPU= 16.117
  12:08:21 STEP 2657 H=2657:00 +CPU= 15.289
  12:08:46 STEP 2658 H=2658:00 +CPU= 24.363
  12:09:01 STEP 2659 H=2659:00 +CPU= 15.480
  12:09:18 STEP 2660 H=2660:00 +CPU= 16.442
  12:09:42 STEP 2661 H=2661:00 +CPU= 24.041
  12:09:58 STEP 2662 H=2662:00 +CPU= 15.975
  12:10:15 STEP 2663 H=2663:00 +CPU= 16.353
  12:10:44 STEP 2664 H=2664:00 +CPU= 28.579
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002664
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002664
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002664
  12:11:00 STEP 2665 H=2665:00 +CPU= 15.549
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002664
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_110.zip
Uploading the intermediate file: upload_file_110.zip
  12:11:15 STEP 2666 H=2666:00 +CPU= 15.439
  12:11:40 STEP 2667 H=2667:00 +CPU= 24.634
  12:11:56 STEP 2668 H=2668:00 +CPU= 15.932
  12:12:12 STEP 2669 H=2669:00 +CPU= 15.754
  12:12:36 STEP 2670 H=2670:00 +CPU= 23.885
  12:12:52 STEP 2671 H=2671:00 +CPU= 15.530
  12:13:08 STEP 2672 H=2672:00 +CPU= 15.460
  12:13:33 STEP 2673 H=2673:00 +CPU= 24.366
  12:13:48 STEP 2674 H=2674:00 +CPU= 15.487
  12:14:04 STEP 2675 H=2675:00 +CPU= 15.261
  12:14:32 STEP 2676 H=2676:00 +CPU= 28.406
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002676
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002676
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002676
  12:14:48 STEP 2677 H=2677:00 +CPU= 15.377
  12:15:03 STEP 2678 H=2678:00 +CPU= 15.334
  12:15:28 STEP 2679 H=2679:00 +CPU= 24.316
  12:15:44 STEP 2680 H=2680:00 +CPU= 16.046
  12:16:01 STEP 2681 H=2681:00 +CPU= 16.232
  12:16:25 STEP 2682 H=2682:00 +CPU= 24.383
  12:16:41 STEP 2683 H=2683:00 +CPU= 15.648
  12:16:58 STEP 2684 H=2684:00 +CPU= 16.305
  12:17:22 STEP 2685 H=2685:00 +CPU= 24.475
  12:17:39 STEP 2686 H=2686:00 +CPU= 16.531
  12:17:56 STEP 2687 H=2687:00 +CPU= 16.228
  12:18:25 STEP 2688 H=2688:00 +CPU= 29.001
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002688
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002688
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002688
  12:18:41 STEP 2689 H=2689:00 +CPU= 15.876
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002688
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_111.zip
Uploading the intermediate file: upload_file_111.zip
  12:18:57 STEP 2690 H=2690:00 +CPU= 15.864
  12:19:22 STEP 2691 H=2691:00 +CPU= 24.635
  12:19:38 STEP 2692 H=2692:00 +CPU= 16.115
  12:19:54 STEP 2693 H=2693:00 +CPU= 15.623
  12:20:20 STEP 2694 H=2694:00 +CPU= 25.581
  12:20:36 STEP 2695 H=2695:00 +CPU= 15.556
  12:20:52 STEP 2696 H=2696:00 +CPU= 15.887
  12:21:16 STEP 2697 H=2697:00 +CPU= 24.147
  12:21:33 STEP 2698 H=2698:00 +CPU= 16.285
  12:21:48 STEP 2699 H=2699:00 +CPU= 15.389
  12:22:15 STEP 2700 H=2700:00 +CPU= 26.879
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002700
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002700
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002700
  12:22:31 STEP 2701 H=2701:00 +CPU= 15.627
  12:22:48 STEP 2702 H=2702:00 +CPU= 16.407
  12:23:13 STEP 2703 H=2703:00 +CPU= 24.959
  12:23:30 STEP 2704 H=2704:00 +CPU= 16.325
  12:23:45 STEP 2705 H=2705:00 +CPU= 15.520
  12:24:11 STEP 2706 H=2706:00 +CPU= 25.495
  12:24:28 STEP 2707 H=2707:00 +CPU= 16.385
  12:24:44 STEP 2708 H=2708:00 +CPU= 15.934
  12:25:09 STEP 2709 H=2709:00 +CPU= 24.969
  12:25:25 STEP 2710 H=2710:00 +CPU= 15.783
  12:25:41 STEP 2711 H=2711:00 +CPU= 15.995
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_1.21_x86_64-pc-linux-gnu
(argv1) start_date: 2019110100
(argv2) exptid: hnw4
(argv3) unique_member_id: 0189
(argv4) batchid: 993
(argv5) wuid: 12215031
(argv6) fclen: 123
(argv7) app_name: oifs_43r3
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/8
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3
version: 1.21
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/8/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/8/oifs_43r3_app_1.21_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_095c0384783f04adfce0982f35dab53b to: /var/lib/boinc/slots/8/oifs_43r3_0189_2019110100_123_993_12215031.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/8/oifs_43r3_0189_2019110100_123_993_12215031.zip
ic_ancil_file: ic_ancil_12215031
ifsdata_file: ifsdata_12215031
climate_data_file: clim_data_12215031
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_35f49e611032e72440e50c216fd6de14 to: /var/lib/boinc/slots/8/ic_ancil_12215031.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/8/ic_ancil_12215031.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8eb38ede50bdfe360fca2d3db4100af0 to: /var/lib/boinc/slots/8/ifsdata/ifsdata_12215031.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/8/ifsdata/ifsdata_12215031.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/8/159l_2/clim_data_12215031.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/8/159l_2/clim_data_12215031.zip
Checking for progress XML file: /var/lib/boinc/slots/8/progress_file_12215031.xml
Opened progress file ok : /var/lib/boinc/slots/8/progress_file_12215031.xml
-- Model is restarting --
Adjusting last_iter, 2712, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/8/progress_file_12215031.xml
last_cpu_time: 58636
upload_file_number: 112
last_iter: 2713
last_upload: 9680400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729
The child process has been launched with process id: 169377
Executing the command: /var/lib/boinc/slots/8/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2091] fp = 0x3124020
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [process_options@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [ignore_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [restore_default_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [trapfpe_treatment@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x810730 (old at 0x1ce8440)
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [signal_drhook_init@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x810730 (old at (nil))
[EC_DRHOOK:Babylon5.RhiOwGwil:1:1:169377:169377] [20230303:123029:1677846629:0.000] [catch_signals@/home/glenn/github/gc_oifs43r3/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_112.zip
Uploading the intermediate file: upload_file_112.zip
  12:30:59 STEP 2712 H=2712:00 +CPU= 27.521
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002712
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002712
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002712
  12:31:14 STEP 2713 H=2713:00 +CPU= 15.043
  12:31:31 STEP 2714 H=2714:00 +CPU= 15.933
  12:31:55 STEP 2715 H=2715:00 +CPU= 24.387
  12:32:11 STEP 2716 H=2716:00 +CPU= 15.867
  12:32:27 STEP 2717 H=2717:00 +CPU= 15.669
  12:32:52 STEP 2718 H=2718:00 +CPU= 24.492
  12:33:08 STEP 2719 H=2719:00 +CPU= 15.646
  12:33:24 STEP 2720 H=2720:00 +CPU= 16.167
  12:33:49 STEP 2721 H=2721:00 +CPU= 24.553
  12:34:05 STEP 2722 H=2722:00 +CPU= 15.963
  12:34:21 STEP 2723 H=2723:00 +CPU= 15.728
  12:34:49 STEP 2724 H=2724:00 +CPU= 27.438
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002724
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002724
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002724
  12:35:05 STEP 2725 H=2725:00 +CPU= 15.723
  12:35:21 STEP 2726 H=2726:00 +CPU= 16.003
  12:35:46 STEP 2727 H=2727:00 +CPU= 24.373
  12:36:02 STEP 2728 H=2728:00 +CPU= 16.575
  12:36:18 STEP 2729 H=2729:00 +CPU= 15.605
  12:36:43 STEP 2730 H=2730:00 +CPU= 24.889
  12:37:00 STEP 2731 H=2731:00 +CPU= 15.858
  12:37:16 STEP 2732 H=2732:00 +CPU= 16.307
  12:37:41 STEP 2733 H=2733:00 +CPU= 24.314
  12:37:57 STEP 2734 H=2734:00 +CPU= 15.842
  12:38:13 STEP 2735 H=2735:00 +CPU= 16.288
  12:38:42 STEP 2736 H=2736:00 +CPU= 28.700
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002736
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002736
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002736
  12:38:58 STEP 2737 H=2737:00 +CPU= 15.717
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002736
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_113.zip
Uploading the intermediate file: upload_file_113.zip
  12:39:15 STEP 2738 H=2738:00 +CPU= 16.173
  12:39:40 STEP 2739 H=2739:00 +CPU= 25.039
  12:39:56 STEP 2740 H=2740:00 +CPU= 15.819
  12:40:12 STEP 2741 H=2741:00 +CPU= 16.046
  12:40:38 STEP 2742 H=2742:00 +CPU= 25.073
  12:40:53 STEP 2743 H=2743:00 +CPU= 15.442
  12:41:10 STEP 2744 H=2744:00 +CPU= 16.437
  12:41:35 STEP 2745 H=2745:00 +CPU= 24.433
  12:41:51 STEP 2746 H=2746:00 +CPU= 16.303
  12:42:07 STEP 2747 H=2747:00 +CPU= 15.932
  12:42:34 STEP 2748 H=2748:00 +CPU= 26.101
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002748
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002748
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002748
  12:42:50 STEP 2749 H=2749:00 +CPU= 15.656
  12:43:06 STEP 2750 H=2750:00 +CPU= 16.143
  12:43:30 STEP 2751 H=2751:00 +CPU= 23.898
  12:43:46 STEP 2752 H=2752:00 +CPU= 15.973
  12:44:02 STEP 2753 H=2753:00 +CPU= 15.568
  12:44:27 STEP 2754 H=2754:00 +CPU= 24.683
  12:44:42 STEP 2755 H=2755:00 +CPU= 15.328
  12:44:58 STEP 2756 H=2756:00 +CPU= 15.461
  12:45:23 STEP 2757 H=2757:00 +CPU= 24.474
  12:45:39 STEP 2758 H=2758:00 +CPU= 15.786
  12:45:55 STEP 2759 H=2759:00 +CPU= 16.032
  12:46:24 STEP 2760 H=2760:00 +CPU= 28.557
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002760
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002760
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002760
  12:46:40 STEP 2761 H=2761:00 +CPU= 15.835
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002760
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_114.zip
Uploading the intermediate file: upload_file_114.zip
  12:46:56 STEP 2762 H=2762:00 +CPU= 15.430
  12:47:20 STEP 2763 H=2763:00 +CPU= 24.006
  12:47:37 STEP 2764 H=2764:00 +CPU= 16.678
  12:47:53 STEP 2765 H=2765:00 +CPU= 15.665
  12:48:18 STEP 2766 H=2766:00 +CPU= 25.630
  12:48:34 STEP 2767 H=2767:00 +CPU= 15.570
  12:48:50 STEP 2768 H=2768:00 +CPU= 15.872
  12:49:15 STEP 2769 H=2769:00 +CPU= 24.003
  12:49:31 STEP 2770 H=2770:00 +CPU= 16.533
  12:49:47 STEP 2771 H=2771:00 +CPU= 15.710
  12:50:14 STEP 2772 H=2772:00 +CPU= 26.619
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002772
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002772
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002772
  12:50:31 STEP 2773 H=2773:00 +CPU= 16.354
  12:50:46 STEP 2774 H=2774:00 +CPU= 15.487
  12:51:11 STEP 2775 H=2775:00 +CPU= 24.720
  12:51:28 STEP 2776 H=2776:00 +CPU= 16.030
  12:51:43 STEP 2777 H=2777:00 +CPU= 15.489
  12:52:08 STEP 2778 H=2778:00 +CPU= 24.811
  12:52:24 STEP 2779 H=2779:00 +CPU= 15.827
  12:52:41 STEP 2780 H=2780:00 +CPU= 16.237
  12:53:05 STEP 2781 H=2781:00 +CPU= 24.014
  12:53:21 STEP 2782 H=2782:00 +CPU= 15.452
  12:53:37 STEP 2783 H=2783:00 +CPU= 16.176
  12:54:07 STEP 2784 H=2784:00 +CPU= 29.170
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002784
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002784
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002784
  12:54:22 STEP 2785 H=2785:00 +CPU= 15.458
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002784
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_115.zip
Uploading the intermediate file: upload_file_115.zip
  12:54:38 STEP 2786 H=2786:00 +CPU= 15.919
  12:55:03 STEP 2787 H=2787:00 +CPU= 24.429
  12:55:19 STEP 2788 H=2788:00 +CPU= 15.995
  12:55:35 STEP 2789 H=2789:00 +CPU= 15.725
  12:56:00 STEP 2790 H=2790:00 +CPU= 24.744
  12:56:16 STEP 2791 H=2791:00 +CPU= 15.955
  12:56:33 STEP 2792 H=2792:00 +CPU= 16.242
  12:56:58 STEP 2793 H=2793:00 +CPU= 24.931
  12:57:14 STEP 2794 H=2794:00 +CPU= 16.071
  12:57:30 STEP 2795 H=2795:00 +CPU= 15.567
  12:57:57 STEP 2796 H=2796:00 +CPU= 26.913
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002796
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002796
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002796
  12:58:13 STEP 2797 H=2797:00 +CPU= 15.791
  12:58:29 STEP 2798 H=2798:00 +CPU= 15.951
  12:58:54 STEP 2799 H=2799:00 +CPU= 24.358
  12:59:10 STEP 2800 H=2800:00 +CPU= 15.897
  12:59:26 STEP 2801 H=2801:00 +CPU= 15.561
  12:59:51 STEP 2802 H=2802:00 +CPU= 24.704
  13:00:07 STEP 2803 H=2803:00 +CPU= 15.716
  13:00:23 STEP 2804 H=2804:00 +CPU= 16.220
  13:00:48 STEP 2805 H=2805:00 +CPU= 24.541
  13:01:03 STEP 2806 H=2806:00 +CPU= 15.368
  13:01:20 STEP 2807 H=2807:00 +CPU= 16.205
  13:01:49 STEP 2808 H=2808:00 +CPU= 28.423
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002808
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002808
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002808
  13:02:04 STEP 2809 H=2809:00 +CPU= 15.355
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002808
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_116.zip
Uploading the intermediate file: upload_file_116.zip
  13:02:21 STEP 2810 H=2810:00 +CPU= 16.382
  13:02:46 STEP 2811 H=2811:00 +CPU= 24.922
  13:03:02 STEP 2812 H=2812:00 +CPU= 16.088
  13:03:18 STEP 2813 H=2813:00 +CPU= 15.675
  13:03:43 STEP 2814 H=2814:00 +CPU= 24.572
  13:03:59 STEP 2815 H=2815:00 +CPU= 15.701
  13:04:15 STEP 2816 H=2816:00 +CPU= 15.824
  13:04:39 STEP 2817 H=2817:00 +CPU= 24.133
  13:04:56 STEP 2818 H=2818:00 +CPU= 16.025
  13:05:12 STEP 2819 H=2819:00 +CPU= 15.839
  13:05:39 STEP 2820 H=2820:00 +CPU= 27.225
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002820
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002820
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002820
  13:05:55 STEP 2821 H=2821:00 +CPU= 15.514
  13:06:11 STEP 2822 H=2822:00 +CPU= 16.205
  13:06:35 STEP 2823 H=2823:00 +CPU= 24.096
  13:06:51 STEP 2824 H=2824:00 +CPU= 15.803
  13:07:07 STEP 2825 H=2825:00 +CPU= 15.320
  13:07:32 STEP 2826 H=2826:00 +CPU= 24.705
  13:07:48 STEP 2827 H=2827:00 +CPU= 15.738
  13:08:04 STEP 2828 H=2828:00 +CPU= 16.164
  13:08:28 STEP 2829 H=2829:00 +CPU= 24.084
  13:08:45 STEP 2830 H=2830:00 +CPU= 16.108
  13:09:01 STEP 2831 H=2831:00 +CPU= 15.803
  13:09:30 STEP 2832 H=2832:00 +CPU= 29.106
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002832
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002832
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002832
  13:09:46 STEP 2833 H=2833:00 +CPU= 15.944
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002832
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_117.zip
Uploading the intermediate file: upload_file_117.zip
  13:10:02 STEP 2834 H=2834:00 +CPU= 15.989
  13:10:27 STEP 2835 H=2835:00 +CPU= 24.663
  13:10:43 STEP 2836 H=2836:00 +CPU= 15.737
  13:10:59 STEP 2837 H=2837:00 +CPU= 15.339
  13:11:23 STEP 2838 H=2838:00 +CPU= 24.154
  13:11:39 STEP 2839 H=2839:00 +CPU= 15.476
  13:11:55 STEP 2840 H=2840:00 +CPU= 15.769
  13:12:19 STEP 2841 H=2841:00 +CPU= 23.567
  13:12:35 STEP 2842 H=2842:00 +CPU= 16.417
  13:12:51 STEP 2843 H=2843:00 +CPU= 15.360
  13:13:17 STEP 2844 H=2844:00 +CPU= 25.844
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002844
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002844
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002844
  13:13:32 STEP 2845 H=2845:00 +CPU= 14.751
  13:13:48 STEP 2846 H=2846:00 +CPU= 15.835
  13:14:12 STEP 2847 H=2847:00 +CPU= 24.072
  13:14:28 STEP 2848 H=2848:00 +CPU= 16.037
  13:14:44 STEP 2849 H=2849:00 +CPU= 15.398
  13:15:09 STEP 2850 H=2850:00 +CPU= 24.768
  13:15:26 STEP 2851 H=2851:00 +CPU= 16.486
  13:15:42 STEP 2852 H=2852:00 +CPU= 16.262
  13:16:06 STEP 2853 H=2853:00 +CPU= 23.845
  13:16:23 STEP 2854 H=2854:00 +CPU= 16.164
  13:16:39 STEP 2855 H=2855:00 +CPU= 15.967
  13:17:08 STEP 2856 H=2856:00 +CPU= 28.838
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002856
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002856
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002856
  13:17:24 STEP 2857 H=2857:00 +CPU= 16.172
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002856
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_118.zip
Uploading the intermediate file: upload_file_118.zip
  13:17:40 STEP 2858 H=2858:00 +CPU= 16.002
  13:18:05 STEP 2859 H=2859:00 +CPU= 24.608
  13:18:22 STEP 2860 H=2860:00 +CPU= 16.264
  13:18:37 STEP 2861 H=2861:00 +CPU= 15.613
  13:19:02 STEP 2862 H=2862:00 +CPU= 24.683
  13:19:19 STEP 2863 H=2863:00 +CPU= 15.874
  13:19:35 STEP 2864 H=2864:00 +CPU= 15.847
  13:19:59 STEP 2865 H=2865:00 +CPU= 24.221
  13:20:15 STEP 2866 H=2866:00 +CPU= 15.896
  13:20:31 STEP 2867 H=2867:00 +CPU= 15.741
  13:20:59 STEP 2868 H=2868:00 +CPU= 27.483
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002868
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002868
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002868
  13:21:15 STEP 2869 H=2869:00 +CPU= 15.693
  13:21:31 STEP 2870 H=2870:00 +CPU= 16.106
  13:21:55 STEP 2871 H=2871:00 +CPU= 24.049
  13:22:11 STEP 2872 H=2872:00 +CPU= 15.573
  13:22:27 STEP 2873 H=2873:00 +CPU= 15.184
  13:22:52 STEP 2874 H=2874:00 +CPU= 24.794
  13:23:07 STEP 2875 H=2875:00 +CPU= 15.164
  13:23:23 STEP 2876 H=2876:00 +CPU= 15.996
  13:23:47 STEP 2877 H=2877:00 +CPU= 23.584
  13:24:03 STEP 2878 H=2878:00 +CPU= 16.155
  13:24:20 STEP 2879 H=2879:00 +CPU= 16.448
  13:24:49 STEP 2880 H=2880:00 +CPU= 28.741
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002880
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002880
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002880
  13:25:04 STEP 2881 H=2881:00 +CPU= 15.237
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002880
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_119.zip
Uploading the intermediate file: upload_file_119.zip
  13:25:22 STEP 2882 H=2882:00 +CPU= 17.033
  13:25:46 STEP 2883 H=2883:00 +CPU= 24.215
  13:26:02 STEP 2884 H=2884:00 +CPU= 16.022
  13:26:18 STEP 2885 H=2885:00 +CPU= 15.779
  13:26:43 STEP 2886 H=2886:00 +CPU= 24.294
  13:26:58 STEP 2887 H=2887:00 +CPU= 15.352
  13:27:14 STEP 2888 H=2888:00 +CPU= 15.703
  13:27:39 STEP 2889 H=2889:00 +CPU= 24.159
  13:27:55 STEP 2890 H=2890:00 +CPU= 16.511
  13:28:11 STEP 2891 H=2891:00 +CPU= 15.546
  13:28:38 STEP 2892 H=2892:00 +CPU= 26.557
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002892
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002892
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002892
  13:28:53 STEP 2893 H=2893:00 +CPU= 15.268
  13:29:10 STEP 2894 H=2894:00 +CPU= 16.352
  13:29:34 STEP 2895 H=2895:00 +CPU= 24.104
  13:29:51 STEP 2896 H=2896:00 +CPU= 16.263
  13:30:07 STEP 2897 H=2897:00 +CPU= 16.522
  13:30:34 STEP 2898 H=2898:00 +CPU= 25.812
  13:30:49 STEP 2899 H=2899:00 +CPU= 15.040
  13:31:05 STEP 2900 H=2900:00 +CPU= 16.345
  13:31:30 STEP 2901 H=2901:00 +CPU= 24.514
  13:31:46 STEP 2902 H=2902:00 +CPU= 15.450
  13:32:02 STEP 2903 H=2903:00 +CPU= 16.472
  13:32:31 STEP 2904 H=2904:00 +CPU= 28.736
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002904
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002904
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002904
  13:32:47 STEP 2905 H=2905:00 +CPU= 14.918
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002892
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002892
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002892
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002904
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_120.zip
Uploading the intermediate file: upload_file_120.zip
  13:33:02 STEP 2906 H=2906:00 +CPU= 15.607
  13:33:27 STEP 2907 H=2907:00 +CPU= 24.085
  13:33:43 STEP 2908 H=2908:00 +CPU= 16.008
  13:33:59 STEP 2909 H=2909:00 +CPU= 15.418
  13:34:24 STEP 2910 H=2910:00 +CPU= 25.270
  13:34:40 STEP 2911 H=2911:00 +CPU= 15.297
  13:34:56 STEP 2912 H=2912:00 +CPU= 15.746
  13:35:20 STEP 2913 H=2913:00 +CPU= 24.154
  13:35:37 STEP 2914 H=2914:00 +CPU= 16.737
  13:35:53 STEP 2915 H=2915:00 +CPU= 15.508
  13:36:19 STEP 2916 H=2916:00 +CPU= 26.203
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002916
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002916
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002916
  13:36:35 STEP 2917 H=2917:00 +CPU= 15.513
  13:36:51 STEP 2918 H=2918:00 +CPU= 15.908
  13:37:15 STEP 2919 H=2919:00 +CPU= 24.146
  13:37:32 STEP 2920 H=2920:00 +CPU= 16.099
  13:37:47 STEP 2921 H=2921:00 +CPU= 15.011
  13:38:11 STEP 2922 H=2922:00 +CPU= 24.331
  13:38:27 STEP 2923 H=2923:00 +CPU= 15.666
  13:38:43 STEP 2924 H=2924:00 +CPU= 15.864
  13:39:07 STEP 2925 H=2925:00 +CPU= 23.870
  13:39:23 STEP 2926 H=2926:00 +CPU= 15.795
  13:39:39 STEP 2927 H=2927:00 +CPU= 15.845
  13:40:08 STEP 2928 H=2928:00 +CPU= 28.628
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002928
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002928
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002928
  13:40:24 STEP 2929 H=2929:00 +CPU= 15.464
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002916
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002916
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002916
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002928
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_121.zip
Uploading the intermediate file: upload_file_121.zip
  13:40:40 STEP 2930 H=2930:00 +CPU= 15.858
Uploading trickle at timestep: 10544400
  13:41:05 STEP 2931 H=2931:00 +CPU= 24.376
  13:41:21 STEP 2932 H=2932:00 +CPU= 15.939
  13:41:36 STEP 2933 H=2933:00 +CPU= 15.280
  13:42:01 STEP 2934 H=2934:00 +CPU= 24.550
  13:42:16 STEP 2935 H=2935:00 +CPU= 15.330
  13:42:32 STEP 2936 H=2936:00 +CPU= 15.871
  13:42:57 STEP 2937 H=2937:00 +CPU= 23.801
  13:43:13 STEP 2938 H=2938:00 +CPU= 15.872
  13:43:28 STEP 2939 H=2939:00 +CPU= 15.448
  13:43:55 STEP 2940 H=2940:00 +CPU= 26.219
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002940
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002940
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002940
  13:44:11 STEP 2941 H=2941:00 +CPU= 15.699
  13:44:28 STEP 2942 H=2942:00 +CPU= 16.932
  13:44:53 STEP 2943 H=2943:00 +CPU= 24.962
  13:45:09 STEP 2944 H=2944:00 +CPU= 15.712
  13:45:25 STEP 2945 H=2945:00 +CPU= 15.704
  13:45:50 STEP 2946 H=2946:00 +CPU= 24.656
  13:46:05 STEP 2947 H=2947:00 +CPU= 15.311
  13:46:21 STEP 2948 H=2948:00 +CPU= 15.767
  13:46:45 STEP 2949 H=2949:00 +CPU= 23.604
  13:47:01 STEP 2950 H=2950:00 +CPU= 15.912
  13:47:16 STEP 2951 H=2951:00 +CPU= 15.098
  13:47:44 STEP 2952 H=2952:00 +CPU= 27.428
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01

 STEPO, YSPSDT_AR1     (  1,...), SP NORMS 2D: 4.865283731608E-01 1.865286143845E-01 6.362598939121E-02
 STEPO, YSPSDT_AR1     (SUM,...), SP NORMS 2D: 5.260318722649E-01
 STEPO, YGPSDT         (  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.48989191816039E-02 -1.72994898449055E+00  1.89761310057317E+00
 STEPO, YGPSDT[clipped](  1,...), Grid Point Norms
  FIELD LEV        AVERAGE               MINIMUM               MAXIMUM       
    101  60  3.19925486346338E-02 -1.00000000000000E+00  1.00000000000000E+00
  MAXGPFV : MAX. VALUE =    1.04868506414152     
  MAXGPFV : MAX. VALUE =   0.741146852392735     
 NSTEP =  2952 SCAN2M_HPOS  P
 IO-STREAM SETUP - IOTYPE =           2  NUMIOPROCS =           1  CPATH =
 ICMGGhnw4+002952 MODE=a
 IO-STREAM CLOSED - ICMGGhnw4+002952
 MPI-TASK:   1 -    339022666 BYTES IN      8 RECORDS TRANSFERRED IN    0.0004 SECONDS847556.7 Mbytes/s, TOTAL TIME=    0.0069( 5.8%)
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302231500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302233000  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200302234500  !!!!!!!!!!!! 
   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20200303000000  !!!!!!!!!!!! 
 13:47:44 STEP 2952 H=2952:00 +CPU= 27.428
 END CNT3
 NSTEP =  2952 CNT0   000000000  23.713  23.713  24.044   0.986               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         3730/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1     93    0       35       786125     3730 ************    15.1     0.9    14.2     1.4    21.0
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 4634873.3 4634873.3    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 4632663.4 4632663.4     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  261    1673.9    1673.9      9.43      0.00
   9 SPCM     - SPECTRAL COMP.                     240     223.9     223.9      1.16      0.00
  10 SCAN2M - PHYSICS                              241    8489.6    8489.6     44.14      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 20    1031.2    1031.2      0.44      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                 11      13.2      13.2      0.00      0.00
  13 SCAN2M - RADIATION CALC.                       81    8684.8    8684.8     15.18      0.00
  17 GRIDFPOS IN CNT4                               20      35.4      35.4      0.02      0.00
  27 RESTART FILES - WRITING                        10    2062.4    2062.4      0.44      0.00
  28 RESTART FILES - READING                         1    1833.3    1833.3      0.04      0.00
  29 SU4FPOS IN CNT4                                20       0.0       0.0      0.00      0.00
  30 DYNFPOS IN CNT4                                20    1636.1    1636.1      0.71      0.00
  31 POSDDH IN STEPO                                 1       0.7       0.7      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    241    2384.3    2384.3     12.40      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           241    2037.6    2037.6     10.59      0.00
  39 SU0YOMB                                         1     245.9     245.9      0.01      0.00
  51 SCAN2M   - SL COMM. PART 1                    241      10.4      10.4      0.05      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              240     163.9     163.9      0.85      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              240      19.3      19.3      0.10      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                 11       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      826      99.0      99.0      1.76      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       500      94.9      94.9      1.02      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        500      45.4      45.4      0.49      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       826      32.5      32.5      0.58      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      15.1      15.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            826      16.9      16.9      0.30      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            500      14.1      14.1      0.15      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            826      18.9      18.9      0.34      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            500      14.1      14.1      0.15      0.00
 400 GSTATS                                      48965       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 46832       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     4634.9 CPU TIME    4581.8 VECTOR TIME     4581.8


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     2294.0
  *** END CNT0 *** 
------------------------------------------------
Moving to projects directory: /var/lib/boinc/slots/8/ICMGGhnw4+002952
Moving to projects directory: /var/lib/boinc/slots/8/ICMSHhnw4+002952
Moving to projects directory: /var/lib/boinc/slots/8/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc/slots/8/NODE.001_01
Adding to the zip: /var/lib/boinc/slots/8/ifs.stat
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002940
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+001800
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002940
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMSHhnw4+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMUAhnw4+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_12215031/ICMGGhnw4+002940
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_0189_2019110100_123_993_12215031_1_r1633798729_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
13:51:07 (169374): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:55:14 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 10,623,600 63,217 0.0060
09 May 2023 16:55:14 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 10,544,400 62,790 0.0060
09 May 2023 16:55:13 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 7,952,400 48,792 0.0061
09 May 2023 16:53:42 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 4,320,000 27,065 0.0063
09 May 2023 16:53:39 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 3,456,000 21,755 0.0063
09 May 2023 16:53:34 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 2,592,000 16,299 0.0063
09 May 2023 16:53:28 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 1,728,000 10,751 0.0062
09 May 2023 16:53:25 1467458 22317004 oifs_43r3_0189_2019110100_123_993_12215031_1 864,000 5,392 0.0062


©2024 climateprediction.net