climateprediction.net home page
Task 22305117

Task 22305117

Name oifs_43r3_ps_0019_2017050100_123_986_12202663_2
Workunit 12202663
Created 2 Feb 2023, 11:42:16 UTC
Sent 2 Feb 2023, 11:43:09 UTC
Report deadline 3 Apr 2023, 11:43:09 UTC
Received 9 Feb 2023, 6:30:06 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1510311
Run time 2 days 5 hours 55 min 22 sec
CPU time 1 days 22 hours 48 min 40 sec
Validate state Valid
Credit 14,873.04
Device peak FLOPS 3.69 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,538.64 MB
Peak swap size 4,978.07 MB
Peak disk usage 2,827.41 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<stderr_txt>
3/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_102.zip
Uploading the intermediate file: upload_file_102.zip
  18:23:28 STEP 2473 H=2473:00 +CPU= 45.118
  18:24:21 STEP 2474 H=2474:00 +CPU= 47.414
  18:25:43 STEP 2475 H=2475:00 +CPU= 71.776
  18:26:36 STEP 2476 H=2476:00 +CPU= 46.174
  18:27:28 STEP 2477 H=2477:00 +CPU= 46.293
  18:28:50 STEP 2478 H=2478:00 +CPU= 72.141
  18:29:42 STEP 2479 H=2479:00 +CPU= 46.135
  18:30:38 STEP 2480 H=2480:00 +CPU= 48.099
  18:32:00 STEP 2481 H=2481:00 +CPU= 72.669
  18:32:55 STEP 2482 H=2482:00 +CPU= 48.233
  18:33:49 STEP 2483 H=2483:00 +CPU= 47.152
  18:35:20 STEP 2484 H=2484:00 +CPU= 80.650
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002484
  18:36:12 STEP 2485 H=2485:00 +CPU= 46.085
  18:37:07 STEP 2486 H=2486:00 +CPU= 48.001
  18:38:28 STEP 2487 H=2487:00 +CPU= 71.253
  18:39:22 STEP 2488 H=2488:00 +CPU= 47.407
  18:40:14 STEP 2489 H=2489:00 +CPU= 46.028
  18:41:37 STEP 2490 H=2490:00 +CPU= 72.461
  18:42:30 STEP 2491 H=2491:00 +CPU= 46.482
  18:43:24 STEP 2492 H=2492:00 +CPU= 47.374
  18:44:46 STEP 2493 H=2493:00 +CPU= 72.683
  18:45:41 STEP 2494 H=2494:00 +CPU= 48.544
  18:46:36 STEP 2495 H=2495:00 +CPU= 47.707
  18:48:09 STEP 2496 H=2496:00 +CPU= 82.628
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_103.zip
Uploading the intermediate file: upload_file_103.zip
Uploading trickle at timestep: 8985600
  18:49:01 STEP 2497 H=2497:00 +CPU= 45.666
  18:49:55 STEP 2498 H=2498:00 +CPU= 47.470
  18:51:15 STEP 2499 H=2499:00 +CPU= 70.366
  18:52:09 STEP 2500 H=2500:00 +CPU= 47.602
  18:53:00 STEP 2501 H=2501:00 +CPU= 45.231
  18:54:23 STEP 2502 H=2502:00 +CPU= 72.391
  18:55:16 STEP 2503 H=2503:00 +CPU= 46.901
  18:56:09 STEP 2504 H=2504:00 +CPU= 47.139
  18:57:32 STEP 2505 H=2505:00 +CPU= 72.109
  18:58:27 STEP 2506 H=2506:00 +CPU= 48.421
  18:59:19 STEP 2507 H=2507:00 +CPU= 46.028
  19:00:51 STEP 2508 H=2508:00 +CPU= 80.928
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002508
  19:01:45 STEP 2509 H=2509:00 +CPU= 47.600
  19:02:38 STEP 2510 H=2510:00 +CPU= 47.448
  19:04:01 STEP 2511 H=2511:00 +CPU= 72.784
  19:04:57 STEP 2512 H=2512:00 +CPU= 49.449
  19:05:49 STEP 2513 H=2513:00 +CPU= 45.786
  19:07:13 STEP 2514 H=2514:00 +CPU= 73.656
  19:08:06 STEP 2515 H=2515:00 +CPU= 46.581
  19:08:59 STEP 2516 H=2516:00 +CPU= 46.578
  19:10:21 STEP 2517 H=2517:00 +CPU= 72.643
  19:11:18 STEP 2518 H=2518:00 +CPU= 49.614
  19:12:12 STEP 2519 H=2519:00 +CPU= 47.849
  19:13:47 STEP 2520 H=2520:00 +CPU= 83.629
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_104.zip
Uploading the intermediate file: upload_file_104.zip
  19:14:40 STEP 2521 H=2521:00 +CPU= 46.777
  19:15:33 STEP 2522 H=2522:00 +CPU= 46.180
  19:16:55 STEP 2523 H=2523:00 +CPU= 71.509
  19:17:50 STEP 2524 H=2524:00 +CPU= 47.642
  19:18:41 STEP 2525 H=2525:00 +CPU= 44.975
  19:20:04 STEP 2526 H=2526:00 +CPU= 73.232
  19:20:57 STEP 2527 H=2527:00 +CPU= 46.162
  19:21:51 STEP 2528 H=2528:00 +CPU= 47.115
  19:23:15 STEP 2529 H=2529:00 +CPU= 73.946
  19:24:11 STEP 2530 H=2530:00 +CPU= 48.550
  19:25:03 STEP 2531 H=2531:00 +CPU= 45.887
  19:26:35 STEP 2532 H=2532:00 +CPU= 81.469
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002532
  19:27:28 STEP 2533 H=2533:00 +CPU= 46.324
  19:28:22 STEP 2534 H=2534:00 +CPU= 46.912
  19:29:44 STEP 2535 H=2535:00 +CPU= 71.736
  19:30:38 STEP 2536 H=2536:00 +CPU= 46.917
  19:31:29 STEP 2537 H=2537:00 +CPU= 45.020
  19:32:54 STEP 2538 H=2538:00 +CPU= 74.642
  19:33:46 STEP 2539 H=2539:00 +CPU= 45.231
  19:34:38 STEP 2540 H=2540:00 +CPU= 47.036
  19:36:02 STEP 2541 H=2541:00 +CPU= 73.770
  19:36:54 STEP 2542 H=2542:00 +CPU= 46.630
  19:37:48 STEP 2543 H=2543:00 +CPU= 47.886
  19:39:26 STEP 2544 H=2544:00 +CPU= 85.779
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_105.zip
Uploading the intermediate file: upload_file_105.zip
  19:40:17 STEP 2545 H=2545:00 +CPU= 45.131
  19:41:10 STEP 2546 H=2546:00 +CPU= 46.713
  19:42:33 STEP 2547 H=2547:00 +CPU= 72.851
  19:43:25 STEP 2548 H=2548:00 +CPU= 46.155
  19:44:16 STEP 2549 H=2549:00 +CPU= 44.771
  19:45:42 STEP 2550 H=2550:00 +CPU= 75.048
  19:46:32 STEP 2551 H=2551:00 +CPU= 45.135
  19:47:27 STEP 2552 H=2552:00 +CPU= 47.268
  19:48:52 STEP 2553 H=2553:00 +CPU= 73.833
  19:49:45 STEP 2554 H=2554:00 +CPU= 46.861
  19:50:38 STEP 2555 H=2555:00 +CPU= 46.895
  19:52:11 STEP 2556 H=2556:00 +CPU= 81.495
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002556
  19:53:02 STEP 2557 H=2557:00 +CPU= 45.007
  19:53:56 STEP 2558 H=2558:00 +CPU= 47.606
  19:55:20 STEP 2559 H=2559:00 +CPU= 73.894
  19:56:14 STEP 2560 H=2560:00 +CPU= 47.165
  19:57:06 STEP 2561 H=2561:00 +CPU= 45.616
  19:58:31 STEP 2562 H=2562:00 +CPU= 74.755
  19:59:22 STEP 2563 H=2563:00 +CPU= 45.190
  20:00:17 STEP 2564 H=2564:00 +CPU= 48.472
  20:01:41 STEP 2565 H=2565:00 +CPU= 73.342
  20:02:34 STEP 2566 H=2566:00 +CPU= 47.031
  20:03:28 STEP 2567 H=2567:00 +CPU= 48.230
  20:05:04 STEP 2568 H=2568:00 +CPU= 83.975
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:05:56 STEP 2569 H=2569:00 +CPU= 45.065
  20:06:50 STEP 2570 H=2570:00 +CPU= 47.406
  20:08:11 STEP 2571 H=2571:00 +CPU= 71.610
  20:09:04 STEP 2572 H=2572:00 +CPU= 46.525
  20:09:56 STEP 2573 H=2573:00 +CPU= 46.000
  20:11:19 STEP 2574 H=2574:00 +CPU= 72.947
  20:12:12 STEP 2575 H=2575:00 +CPU= 46.207
  20:13:07 STEP 2576 H=2576:00 +CPU= 48.074
  20:14:28 STEP 2577 H=2577:00 +CPU= 72.046
  20:15:24 STEP 2578 H=2578:00 +CPU= 48.558
  20:16:15 STEP 2579 H=2579:00 +CPU= 45.445
  20:17:47 STEP 2580 H=2580:00 +CPU= 81.121
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002580
  20:18:41 STEP 2581 H=2581:00 +CPU= 47.310
  20:19:34 STEP 2582 H=2582:00 +CPU= 47.303
  20:20:55 STEP 2583 H=2583:00 +CPU= 72.216
  20:21:50 STEP 2584 H=2584:00 +CPU= 48.664
  20:22:41 STEP 2585 H=2585:00 +CPU= 44.854
  20:24:06 STEP 2586 H=2586:00 +CPU= 74.404
  20:24:59 STEP 2587 H=2587:00 +CPU= 46.983
  20:25:52 STEP 2588 H=2588:00 +CPU= 47.051
  20:27:15 STEP 2589 H=2589:00 +CPU= 72.941
  20:28:10 STEP 2590 H=2590:00 +CPU= 49.144
  20:29:04 STEP 2591 H=2591:00 +CPU= 47.182
  20:30:39 STEP 2592 H=2592:00 +CPU= 84.049
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:31:32 STEP 2593 H=2593:00 +CPU= 46.955
  20:32:24 STEP 2594 H=2594:00 +CPU= 45.863
  20:33:47 STEP 2595 H=2595:00 +CPU= 71.675
  20:34:41 STEP 2596 H=2596:00 +CPU= 48.141
  20:35:32 STEP 2597 H=2597:00 +CPU= 44.656
  20:36:55 STEP 2598 H=2598:00 +CPU= 73.007
  20:37:49 STEP 2599 H=2599:00 +CPU= 47.340
  20:38:42 STEP 2600 H=2600:00 +CPU= 47.474
  20:40:04 STEP 2601 H=2601:00 +CPU= 72.383
  20:41:01 STEP 2602 H=2602:00 +CPU= 49.389
  20:41:53 STEP 2603 H=2603:00 +CPU= 45.821
  20:43:26 STEP 2604 H=2604:00 +CPU= 80.596
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002604
  20:44:20 STEP 2605 H=2605:00 +CPU= 48.381
  20:45:13 STEP 2606 H=2606:00 +CPU= 46.647
  20:46:34 STEP 2607 H=2607:00 +CPU= 71.767
  20:47:29 STEP 2608 H=2608:00 +CPU= 48.175
  20:48:19 STEP 2609 H=2609:00 +CPU= 44.888
  20:49:42 STEP 2610 H=2610:00 +CPU= 73.749
  20:50:37 STEP 2611 H=2611:00 +CPU= 46.915
  20:51:31 STEP 2612 H=2612:00 +CPU= 47.180
  20:52:52 STEP 2613 H=2613:00 +CPU= 71.570
  20:53:48 STEP 2614 H=2614:00 +CPU= 48.767
  20:54:42 STEP 2615 H=2615:00 +CPU= 47.726
  20:56:16 STEP 2616 H=2616:00 +CPU= 82.826
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_108.zip
Uploading the intermediate file: upload_file_108.zip
  20:57:10 STEP 2617 H=2617:00 +CPU= 47.244
  20:58:02 STEP 2618 H=2618:00 +CPU= 46.445
  20:59:22 STEP 2619 H=2619:00 +CPU= 70.912
  21:00:17 STEP 2620 H=2620:00 +CPU= 48.041
  21:01:09 STEP 2621 H=2621:00 +CPU= 45.758
  21:02:30 STEP 2622 H=2622:00 +CPU= 71.332
  21:03:24 STEP 2623 H=2623:00 +CPU= 47.286
  21:04:19 STEP 2624 H=2624:00 +CPU= 48.019
  21:05:40 STEP 2625 H=2625:00 +CPU= 70.572
  21:06:36 STEP 2626 H=2626:00 +CPU= 49.258
  21:07:30 STEP 2627 H=2627:00 +CPU= 47.542
  21:08:59 STEP 2628 H=2628:00 +CPU= 78.864
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002628
  21:09:52 STEP 2629 H=2629:00 +CPU= 47.420
  21:10:48 STEP 2630 H=2630:00 +CPU= 48.946
  21:12:06 STEP 2631 H=2631:00 +CPU= 69.607
  21:13:01 STEP 2632 H=2632:00 +CPU= 48.229
  21:13:54 STEP 2633 H=2633:00 +CPU= 46.878
  21:15:15 STEP 2634 H=2634:00 +CPU= 71.568
  21:16:08 STEP 2635 H=2635:00 +CPU= 46.992
  21:17:04 STEP 2636 H=2636:00 +CPU= 48.660
  21:18:24 STEP 2637 H=2637:00 +CPU= 70.654
  21:19:19 STEP 2638 H=2638:00 +CPU= 48.242
  21:20:15 STEP 2639 H=2639:00 +CPU= 49.076
  21:21:48 STEP 2640 H=2640:00 +CPU= 82.422
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_109.zip
Uploading the intermediate file: upload_file_109.zip
  21:22:41 STEP 2641 H=2641:00 +CPU= 46.497
  21:23:36 STEP 2642 H=2642:00 +CPU= 48.184
  21:24:55 STEP 2643 H=2643:00 +CPU= 70.203
  21:25:49 STEP 2644 H=2644:00 +CPU= 47.157
  21:26:41 STEP 2645 H=2645:00 +CPU= 46.297
  21:28:02 STEP 2646 H=2646:00 +CPU= 70.877
  21:28:55 STEP 2647 H=2647:00 +CPU= 46.933
  21:29:51 STEP 2648 H=2648:00 +CPU= 48.676
  21:31:11 STEP 2649 H=2649:00 +CPU= 70.276
  21:32:07 STEP 2650 H=2650:00 +CPU= 49.123
  21:33:02 STEP 2651 H=2651:00 +CPU= 48.153
  21:34:32 STEP 2652 H=2652:00 +CPU= 79.233
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002652
  21:35:26 STEP 2653 H=2653:00 +CPU= 46.841
  21:36:21 STEP 2654 H=2654:00 +CPU= 48.538
  21:37:40 STEP 2655 H=2655:00 +CPU= 68.840
  21:38:35 STEP 2656 H=2656:00 +CPU= 48.340
  21:39:29 STEP 2657 H=2657:00 +CPU= 46.774
  21:40:50 STEP 2658 H=2658:00 +CPU= 71.883
  21:41:44 STEP 2659 H=2659:00 +CPU= 47.028
  21:42:40 STEP 2660 H=2660:00 +CPU= 49.167
  21:44:01 STEP 2661 H=2661:00 +CPU= 71.382
  21:44:56 STEP 2662 H=2662:00 +CPU= 48.239
  21:45:52 STEP 2663 H=2663:00 +CPU= 49.073
  21:47:24 STEP 2664 H=2664:00 +CPU= 81.206
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_110.zip
Uploading the intermediate file: upload_file_110.zip
  21:48:18 STEP 2665 H=2665:00 +CPU= 47.311
  21:49:13 STEP 2666 H=2666:00 +CPU= 48.249
  21:50:32 STEP 2667 H=2667:00 +CPU= 69.553
  21:51:26 STEP 2668 H=2668:00 +CPU= 47.590
  21:52:20 STEP 2669 H=2669:00 +CPU= 47.032
  21:53:43 STEP 2670 H=2670:00 +CPU= 73.118
  21:54:34 STEP 2671 H=2671:00 +CPU= 45.050
  21:55:30 STEP 2672 H=2672:00 +CPU= 49.296
  21:56:53 STEP 2673 H=2673:00 +CPU= 72.720
  21:57:46 STEP 2674 H=2674:00 +CPU= 46.390
  21:58:40 STEP 2675 H=2675:00 +CPU= 47.510
  22:00:12 STEP 2676 H=2676:00 +CPU= 80.146
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002676
  22:01:04 STEP 2677 H=2677:00 +CPU= 45.473
  22:02:00 STEP 2678 H=2678:00 +CPU= 49.113
  22:03:23 STEP 2679 H=2679:00 +CPU= 73.018
  22:04:15 STEP 2680 H=2680:00 +CPU= 46.064
  22:05:09 STEP 2681 H=2681:00 +CPU= 46.919
  22:06:33 STEP 2682 H=2682:00 +CPU= 74.278
  22:07:22 STEP 2683 H=2683:00 +CPU= 43.275
  22:08:18 STEP 2684 H=2684:00 +CPU= 48.286
  22:09:42 STEP 2685 H=2685:00 +CPU= 74.302
  22:10:33 STEP 2686 H=2686:00 +CPU= 45.181
  22:11:29 STEP 2687 H=2687:00 +CPU= 49.173
  22:13:06 STEP 2688 H=2688:00 +CPU= 84.403
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_111.zip
Uploading the intermediate file: upload_file_111.zip
  22:13:56 STEP 2689 H=2689:00 +CPU= 44.059
  22:14:50 STEP 2690 H=2690:00 +CPU= 47.588
  22:16:13 STEP 2691 H=2691:00 +CPU= 72.730
  22:17:04 STEP 2692 H=2692:00 +CPU= 44.255
  22:17:58 STEP 2693 H=2693:00 +CPU= 46.965
  22:19:24 STEP 2694 H=2694:00 +CPU= 75.617
  22:20:14 STEP 2695 H=2695:00 +CPU= 43.694
  22:21:10 STEP 2696 H=2696:00 +CPU= 48.958
  22:22:34 STEP 2697 H=2697:00 +CPU= 74.189
  22:23:26 STEP 2698 H=2698:00 +CPU= 45.934
  22:24:21 STEP 2699 H=2699:00 +CPU= 47.543
  22:25:54 STEP 2700 H=2700:00 +CPU= 81.727
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002700
  22:26:43 STEP 2701 H=2701:00 +CPU= 43.542
  22:27:39 STEP 2702 H=2702:00 +CPU= 48.672
  22:29:03 STEP 2703 H=2703:00 +CPU= 72.994
  22:29:53 STEP 2704 H=2704:00 +CPU= 44.800
  22:30:47 STEP 2705 H=2705:00 +CPU= 46.840
  22:32:11 STEP 2706 H=2706:00 +CPU= 74.016
  22:33:01 STEP 2707 H=2707:00 +CPU= 43.597
  22:33:55 STEP 2708 H=2708:00 +CPU= 48.093
  22:35:18 STEP 2709 H=2709:00 +CPU= 73.505
  22:36:10 STEP 2710 H=2710:00 +CPU= 45.168
  22:37:06 STEP 2711 H=2711:00 +CPU= 48.858
  22:38:43 STEP 2712 H=2712:00 +CPU= 86.230
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_112.zip
Uploading the intermediate file: upload_file_112.zip
  22:39:32 STEP 2713 H=2713:00 +CPU= 43.286
  22:40:26 STEP 2714 H=2714:00 +CPU= 47.361
  22:41:47 STEP 2715 H=2715:00 +CPU= 71.916
  22:42:37 STEP 2716 H=2716:00 +CPU= 44.552
  22:43:31 STEP 2717 H=2717:00 +CPU= 47.145
  22:44:55 STEP 2718 H=2718:00 +CPU= 75.082
  22:45:45 STEP 2719 H=2719:00 +CPU= 43.801
  22:46:40 STEP 2720 H=2720:00 +CPU= 48.346
  22:48:05 STEP 2721 H=2721:00 +CPU= 73.888
  22:48:58 STEP 2722 H=2722:00 +CPU= 46.067
  22:49:50 STEP 2723 H=2723:00 +CPU= 45.459
  22:51:24 STEP 2724 H=2724:00 +CPU= 82.117
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002724
  22:52:14 STEP 2725 H=2725:00 +CPU= 44.381
  22:53:09 STEP 2726 H=2726:00 +CPU= 47.380
  22:54:33 STEP 2727 H=2727:00 +CPU= 73.978
  22:55:26 STEP 2728 H=2728:00 +CPU= 46.671
  22:56:17 STEP 2729 H=2729:00 +CPU= 44.887
  22:57:43 STEP 2730 H=2730:00 +CPU= 75.196
  22:58:37 STEP 2731 H=2731:00 +CPU= 46.899
  22:59:29 STEP 2732 H=2732:00 +CPU= 45.846
  23:00:54 STEP 2733 H=2733:00 +CPU= 74.759
  23:01:49 STEP 2734 H=2734:00 +CPU= 48.142
  23:02:41 STEP 2735 H=2735:00 +CPU= 45.778
  23:04:18 STEP 2736 H=2736:00 +CPU= 84.642
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_113.zip
Uploading the intermediate file: upload_file_113.zip
Uploading trickle at timestep: 9849600
  23:05:10 STEP 2737 H=2737:00 +CPU= 45.528
  23:06:01 STEP 2738 H=2738:00 +CPU= 45.050
  23:07:24 STEP 2739 H=2739:00 +CPU= 72.841
  23:08:17 STEP 2740 H=2740:00 +CPU= 46.889
  23:09:06 STEP 2741 H=2741:00 +CPU= 43.067
  23:10:31 STEP 2742 H=2742:00 +CPU= 74.396
  23:11:24 STEP 2743 H=2743:00 +CPU= 46.816
  23:12:16 STEP 2744 H=2744:00 +CPU= 45.306
  23:13:37 STEP 2745 H=2745:00 +CPU= 72.269
  23:14:32 STEP 2746 H=2746:00 +CPU= 48.118
  23:15:19 STEP 2747 H=2747:00 +CPU= 42.608
  23:16:51 STEP 2748 H=2748:00 +CPU= 80.979
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002748
  23:17:44 STEP 2749 H=2749:00 +CPU= 46.097
  23:18:33 STEP 2750 H=2750:00 +CPU= 43.398
  23:19:54 STEP 2751 H=2751:00 +CPU= 70.830
  23:20:47 STEP 2752 H=2752:00 +CPU= 46.801
  23:21:34 STEP 2753 H=2753:00 +CPU= 42.448
  23:22:56 STEP 2754 H=2754:00 +CPU= 71.608
  23:23:48 STEP 2755 H=2755:00 +CPU= 45.905
  23:24:39 STEP 2756 H=2756:00 +CPU= 45.094
  23:25:58 STEP 2757 H=2757:00 +CPU= 69.863
  23:26:51 STEP 2758 H=2758:00 +CPU= 47.022
  23:27:44 STEP 2759 H=2759:00 +CPU= 45.954
  23:29:17 STEP 2760 H=2760:00 +CPU= 81.277
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_114.zip
Uploading the intermediate file: upload_file_114.zip
  23:30:10 STEP 2761 H=2761:00 +CPU= 46.123
  23:31:00 STEP 2762 H=2762:00 +CPU= 44.808
  23:32:19 STEP 2763 H=2763:00 +CPU= 69.010
  23:33:12 STEP 2764 H=2764:00 +CPU= 47.170
  23:34:01 STEP 2765 H=2765:00 +CPU= 43.310
  23:35:22 STEP 2766 H=2766:00 +CPU= 71.441
  23:36:14 STEP 2767 H=2767:00 +CPU= 45.658
  23:37:07 STEP 2768 H=2768:00 +CPU= 46.667
  23:38:25 STEP 2769 H=2769:00 +CPU= 68.840
  23:39:20 STEP 2770 H=2770:00 +CPU= 47.262
  23:40:10 STEP 2771 H=2771:00 +CPU= 44.764
  23:41:37 STEP 2772 H=2772:00 +CPU= 76.751
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002772
  23:42:29 STEP 2773 H=2773:00 +CPU= 45.638
  23:43:22 STEP 2774 H=2774:00 +CPU= 46.127
  23:44:39 STEP 2775 H=2775:00 +CPU= 68.682
  23:45:33 STEP 2776 H=2776:00 +CPU= 46.673
  23:46:24 STEP 2777 H=2777:00 +CPU= 45.491
  23:47:45 STEP 2778 H=2778:00 +CPU= 69.852
  23:48:37 STEP 2779 H=2779:00 +CPU= 45.941
  23:49:30 STEP 2780 H=2780:00 +CPU= 47.164
  23:50:48 STEP 2781 H=2781:00 +CPU= 68.412
  23:51:42 STEP 2782 H=2782:00 +CPU= 47.509
  23:52:36 STEP 2783 H=2783:00 +CPU= 47.390
  23:54:03 STEP 2784 H=2784:00 +CPU= 77.900
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_115.zip
Uploading the intermediate file: upload_file_115.zip
  23:54:54 STEP 2785 H=2785:00 +CPU= 44.922
  23:55:47 STEP 2786 H=2786:00 +CPU= 46.680
  23:57:05 STEP 2787 H=2787:00 +CPU= 68.126
  23:57:57 STEP 2788 H=2788:00 +CPU= 45.612
  23:58:48 STEP 2789 H=2789:00 +CPU= 45.388
  00:00:08 STEP 2790 H=2790:00 +CPU= 70.320
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:47:16 STEP 2791 H=2791:00 +CPU= 43.206
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  01:10:24 STEP 2792 H=2792:00 +CPU= 43.790
Suspend request received from the BOINC client, suspending the child process
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.704] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4749MB (heap), 4537MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.706] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.706] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.706] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.706] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.706] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4749 MB (maxheap), 4537 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :  MASTER 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :   CNT0 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :    CNT1 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :     CNT2 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :      CNT3 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :       CNT4 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :        STEPO 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :         SCAN2M 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :          GP_MODEL_HEAP 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :           GP_MODEL 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :            CALL_SL_HEAP 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :             CALL_SL 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :              >OMP-CALL_SL     1        (1004) 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :               LAPINEA 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                LARMES 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                 LARCINA 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022804:1675880884:410710.707] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] :                  LASCAW 
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022805:1675880885:410711.707] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:eric-T14:1:1:8926:8926] [20230209:022805:1675880885:410711.707] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
oifs_43r3_model.e  0000000001CE916B  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000081FFF1  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DC9090  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000000DF12CD  lascaw_                   471  lascaw.F90
oifs_43r3_model.e  0000000000DED525  larcina_                  319  larcina.F90
oifs_43r3_model.e  0000000000DEABF8  larmes_                   603  larmes.F90
oifs_43r3_model.e  0000000000DE50FD  lapinea_                  270  lapinea.F90
oifs_43r3_model.e  0000000000D7A4EC  call_sl_                  300  call_sl.F90
oifs_43r3_model.e  0000000000D784FD  call_sl_heap_              94  call_sl_heap.F90
oifs_43r3_model.e  0000000000BF9958  gp_model_                 533  gp_model.F90
oifs_43r3_model.e  00000000012E9482  gp_model_heap_             74  gp_model_heap.F90
oifs_43r3_model.e  0000000000BBC1CE  scan2m_                   535  scan2m.F90
oifs_43r3_model.e  000000000057A664  stepo_                    327  stepo.F90
oifs_43r3_model.e  000000000055DEEC  cnt4_                    1133  cnt4.F90
oifs_43r3_model.e  00000000005471C9  cnt3_                     267  cnt3.F90
oifs_43r3_model.e  0000000000546412  cnt2_                      88  cnt2.F90
oifs_43r3_model.e  0000000000545E28  cnt1_                      92  cnt1.F90
oifs_43r3_model.e  000000000040708D  cnt0_                     146  cnt0.F90
oifs_43r3_model.e  000000000040220F  MAIN__                     96  master.F90
oifs_43r3_model.e  00000000004021A2  Unknown               Unknown  Unknown
oifs_43r3_model.e  0000000001DCA390  Unknown               Unknown  Unknown
oifs_43r3_model.e  000000000040206E  Unknown               Unknown  Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2017050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0019
(argv4) batchid: 986
(argv5) wuid: 12202663
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/13
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/13/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_e39c498cfd395d4170c0cafa423ef7b5 to: /var/lib/boinc/slots/13/oifs_43r3_ps_0019_2017050100_123_986_12202663.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/13/oifs_43r3_ps_0019_2017050100_123_986_12202663.zip
ic_ancil_file: ic_ancil_12202663
ifsdata_file: ifsdata_12202663
climate_data_file: clim_data_12202663
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_a011e140957ae61a2a2cd5decb5e4c96 to: /var/lib/boinc/slots/13/ic_ancil_12202663.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/13/ic_ancil_12202663.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_1643b4d27c15bbe1e46f7352048ac6e0 to: /var/lib/boinc/slots/13/ifsdata/ifsdata_12202663.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/13/ifsdata/ifsdata_12202663.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/13/159l_2/clim_data_12202663.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/13/159l_2/clim_data_12202663.zip
Checking for progress XML file: /var/lib/boinc/slots/13/progress_file_12202663.xml
Opened progress file ok : /var/lib/boinc/slots/13/progress_file_12202663.xml
-- Model is restarting --
Adjusting last_iter, 2793, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/13/progress_file_12202663.xml
last_cpu_time: 158964
upload_file_number: 116
last_iter: 2783
last_upload: 10022400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494
The child process has been launched with process id: 9177
Executing the command: /var/lib/boinc/slots/13/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:eric-T14:1:1:9177:9177] [20230209:103528:1675910128:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  10:36:35 STEP 2784 H=2784:00 +CPU= 59.015
  10:37:19 STEP 2785 H=2785:00 +CPU= 41.369
  10:38:04 STEP 2786 H=2786:00 +CPU= 42.040
  10:39:14 STEP 2787 H=2787:00 +CPU= 65.386
  10:40:04 STEP 2788 H=2788:00 +CPU= 46.518
  10:40:53 STEP 2789 H=2789:00 +CPU= 45.318
  10:42:09 STEP 2790 H=2790:00 +CPU= 70.608
  10:42:57 STEP 2791 H=2791:00 +CPU= 44.909
  10:43:49 STEP 2792 H=2792:00 +CPU= 47.442
  10:45:04 STEP 2793 H=2793:00 +CPU= 69.527
  10:45:53 STEP 2794 H=2794:00 +CPU= 45.596
  10:46:43 STEP 2795 H=2795:00 +CPU= 45.811
  10:48:07 STEP 2796 H=2796:00 +CPU= 77.816
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002796
  10:48:55 STEP 2797 H=2797:00 +CPU= 44.443
  10:49:46 STEP 2798 H=2798:00 +CPU= 47.562
  10:51:03 STEP 2799 H=2799:00 +CPU= 70.966
  10:51:51 STEP 2800 H=2800:00 +CPU= 44.739
  10:52:40 STEP 2801 H=2801:00 +CPU= 45.833
  10:54:01 STEP 2802 H=2802:00 +CPU= 74.180
  10:54:47 STEP 2803 H=2803:00 +CPU= 43.489
  10:55:41 STEP 2804 H=2804:00 +CPU= 49.148
  10:57:00 STEP 2805 H=2805:00 +CPU= 72.931
  10:57:48 STEP 2806 H=2806:00 +CPU= 45.113
  10:58:41 STEP 2807 H=2807:00 +CPU= 48.986
  11:00:11 STEP 2808 H=2808:00 +CPU= 82.996
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_116.zip
Uploading the intermediate file: upload_file_116.zip
  11:00:59 STEP 2809 H=2809:00 +CPU= 44.005
  11:01:52 STEP 2810 H=2810:00 +CPU= 48.902
  11:03:10 STEP 2811 H=2811:00 +CPU= 72.268
  11:03:59 STEP 2812 H=2812:00 +CPU= 45.583
  11:04:51 STEP 2813 H=2813:00 +CPU= 47.535
  11:06:11 STEP 2814 H=2814:00 +CPU= 73.796
  11:06:58 STEP 2815 H=2815:00 +CPU= 43.726
  11:07:50 STEP 2816 H=2816:00 +CPU= 47.993
  11:09:08 STEP 2817 H=2817:00 +CPU= 71.801
  11:09:56 STEP 2818 H=2818:00 +CPU= 44.786
  11:10:47 STEP 2819 H=2819:00 +CPU= 47.267
  11:12:16 STEP 2820 H=2820:00 +CPU= 81.397
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002820
  11:13:04 STEP 2821 H=2821:00 +CPU= 44.089
  11:13:58 STEP 2822 H=2822:00 +CPU= 50.493
  11:15:18 STEP 2823 H=2823:00 +CPU= 73.329
  11:16:08 STEP 2824 H=2824:00 +CPU= 45.883
  11:16:59 STEP 2825 H=2825:00 +CPU= 47.688
  11:18:20 STEP 2826 H=2826:00 +CPU= 74.706
  11:19:08 STEP 2827 H=2827:00 +CPU= 44.020
  11:20:00 STEP 2828 H=2828:00 +CPU= 48.637
  11:21:19 STEP 2829 H=2829:00 +CPU= 73.403
  11:22:08 STEP 2830 H=2830:00 +CPU= 45.838
  11:23:01 STEP 2831 H=2831:00 +CPU= 49.047
  11:24:33 STEP 2832 H=2832:00 +CPU= 85.113
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_117.zip
Uploading the intermediate file: upload_file_117.zip
  11:25:24 STEP 2833 H=2833:00 +CPU= 46.134
  11:26:18 STEP 2834 H=2834:00 +CPU= 49.845
  11:27:37 STEP 2835 H=2835:00 +CPU= 72.886
  11:28:28 STEP 2836 H=2836:00 +CPU= 47.533
  11:29:20 STEP 2837 H=2837:00 +CPU= 48.561
  11:30:41 STEP 2838 H=2838:00 +CPU= 74.971
  11:31:32 STEP 2839 H=2839:00 +CPU= 47.412
  11:32:27 STEP 2840 H=2840:00 +CPU= 50.134
  11:33:46 STEP 2841 H=2841:00 +CPU= 72.566
  11:34:38 STEP 2842 H=2842:00 +CPU= 48.182
  11:35:30 STEP 2843 H=2843:00 +CPU= 48.066
  11:36:57 STEP 2844 H=2844:00 +CPU= 80.323
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002844
  11:37:48 STEP 2845 H=2845:00 +CPU= 47.466
  11:38:42 STEP 2846 H=2846:00 +CPU= 49.994
  11:39:59 STEP 2847 H=2847:00 +CPU= 71.476
  11:40:51 STEP 2848 H=2848:00 +CPU= 48.751
  11:41:44 STEP 2849 H=2849:00 +CPU= 49.015
  11:43:03 STEP 2850 H=2850:00 +CPU= 72.757
  11:43:55 STEP 2851 H=2851:00 +CPU= 48.186
  11:44:50 STEP 2852 H=2852:00 +CPU= 50.814
  11:46:09 STEP 2853 H=2853:00 +CPU= 72.677
  11:47:03 STEP 2854 H=2854:00 +CPU= 49.631
  11:47:57 STEP 2855 H=2855:00 +CPU= 50.326
  11:49:27 STEP 2856 H=2856:00 +CPU= 83.238
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_118.zip
Uploading the intermediate file: upload_file_118.zip
  11:50:21 STEP 2857 H=2857:00 +CPU= 49.267
  11:51:13 STEP 2858 H=2858:00 +CPU= 48.785
  11:52:31 STEP 2859 H=2859:00 +CPU= 71.584
  11:53:24 STEP 2860 H=2860:00 +CPU= 49.362
  11:54:17 STEP 2861 H=2861:00 +CPU= 49.045
  11:55:37 STEP 2862 H=2862:00 +CPU= 73.812
  11:56:30 STEP 2863 H=2863:00 +CPU= 48.931
  11:57:24 STEP 2864 H=2864:00 +CPU= 50.361
  11:58:42 STEP 2865 H=2865:00 +CPU= 72.252
  11:59:36 STEP 2866 H=2866:00 +CPU= 50.466
  12:00:30 STEP 2867 H=2867:00 +CPU= 49.352
  12:01:55 STEP 2868 H=2868:00 +CPU= 79.314
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002868
  12:02:47 STEP 2869 H=2869:00 +CPU= 48.518
  12:03:40 STEP 2870 H=2870:00 +CPU= 49.165
  12:04:57 STEP 2871 H=2871:00 +CPU= 70.803
  12:05:52 STEP 2872 H=2872:00 +CPU= 50.849
  12:06:45 STEP 2873 H=2873:00 +CPU= 48.681
  12:08:05 STEP 2874 H=2874:00 +CPU= 73.729
  12:08:57 STEP 2875 H=2875:00 +CPU= 48.300
  12:09:51 STEP 2876 H=2876:00 +CPU= 50.191
  12:11:09 STEP 2877 H=2877:00 +CPU= 72.343
  12:12:04 STEP 2878 H=2878:00 +CPU= 50.215
  12:12:59 STEP 2879 H=2879:00 +CPU= 50.708
  12:14:29 STEP 2880 H=2880:00 +CPU= 84.046
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_119.zip
Uploading the intermediate file: upload_file_119.zip
  12:15:22 STEP 2881 H=2881:00 +CPU= 48.541
  12:16:15 STEP 2882 H=2882:00 +CPU= 49.188
  12:17:34 STEP 2883 H=2883:00 +CPU= 72.447
  12:18:28 STEP 2884 H=2884:00 +CPU= 49.809
  12:19:19 STEP 2885 H=2885:00 +CPU= 47.164
  12:20:38 STEP 2886 H=2886:00 +CPU= 73.421
  12:21:30 STEP 2887 H=2887:00 +CPU= 47.661
  12:22:22 STEP 2888 H=2888:00 +CPU= 48.208
  12:23:40 STEP 2889 H=2889:00 +CPU= 72.785
  12:24:34 STEP 2890 H=2890:00 +CPU= 49.534
  12:25:24 STEP 2891 H=2891:00 +CPU= 46.301
  12:26:51 STEP 2892 H=2892:00 +CPU= 80.692
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002892
  12:27:43 STEP 2893 H=2893:00 +CPU= 48.284
  12:28:35 STEP 2894 H=2894:00 +CPU= 48.033
  12:29:54 STEP 2895 H=2895:00 +CPU= 72.390
  12:30:47 STEP 2896 H=2896:00 +CPU= 49.614
  12:31:38 STEP 2897 H=2897:00 +CPU= 46.623
  12:32:58 STEP 2898 H=2898:00 +CPU= 74.069
  12:33:50 STEP 2899 H=2899:00 +CPU= 48.627
  12:34:41 STEP 2900 H=2900:00 +CPU= 47.318
  12:35:59 STEP 2901 H=2901:00 +CPU= 72.194
  12:36:52 STEP 2902 H=2902:00 +CPU= 49.209
  12:37:43 STEP 2903 H=2903:00 +CPU= 47.469
  12:39:14 STEP 2904 H=2904:00 +CPU= 83.992
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_120.zip
Uploading the intermediate file: upload_file_120.zip
  12:40:06 STEP 2905 H=2905:00 +CPU= 47.791
  12:40:57 STEP 2906 H=2906:00 +CPU= 47.829
  12:42:16 STEP 2907 H=2907:00 +CPU= 72.563
  12:43:11 STEP 2908 H=2908:00 +CPU= 50.359
  12:44:01 STEP 2909 H=2909:00 +CPU= 46.584
  12:45:21 STEP 2910 H=2910:00 +CPU= 74.185
  12:46:13 STEP 2911 H=2911:00 +CPU= 47.712
  12:47:05 STEP 2912 H=2912:00 +CPU= 48.101
  12:48:24 STEP 2913 H=2913:00 +CPU= 73.245
  12:49:18 STEP 2914 H=2914:00 +CPU= 50.108
  12:50:09 STEP 2915 H=2915:00 +CPU= 46.781
  12:51:38 STEP 2916 H=2916:00 +CPU= 82.973
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002916
  12:52:30 STEP 2917 H=2917:00 +CPU= 48.204
  12:53:21 STEP 2918 H=2918:00 +CPU= 46.578
  12:54:41 STEP 2919 H=2919:00 +CPU= 74.350
  12:55:35 STEP 2920 H=2920:00 +CPU= 50.069
  12:56:24 STEP 2921 H=2921:00 +CPU= 44.853
  12:57:44 STEP 2922 H=2922:00 +CPU= 74.566
  12:58:37 STEP 2923 H=2923:00 +CPU= 48.717
  12:59:27 STEP 2924 H=2924:00 +CPU= 46.270
  13:00:47 STEP 2925 H=2925:00 +CPU= 73.897
  13:01:41 STEP 2926 H=2926:00 +CPU= 49.515
  13:02:31 STEP 2927 H=2927:00 +CPU= 45.838
  13:04:04 STEP 2928 H=2928:00 +CPU= 85.963
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_121.zip
Uploading the intermediate file: upload_file_121.zip
  13:04:56 STEP 2929 H=2929:00 +CPU= 47.540
  13:05:46 STEP 2930 H=2930:00 +CPU= 45.970
  13:07:06 STEP 2931 H=2931:00 +CPU= 74.072
  13:08:00 STEP 2932 H=2932:00 +CPU= 48.877
  13:08:47 STEP 2933 H=2933:00 +CPU= 44.385
  13:10:09 STEP 2934 H=2934:00 +CPU= 75.813
  13:11:01 STEP 2935 H=2935:00 +CPU= 47.603
  13:11:51 STEP 2936 H=2936:00 +CPU= 46.411
  13:13:11 STEP 2937 H=2937:00 +CPU= 73.900
  13:14:03 STEP 2938 H=2938:00 +CPU= 48.436
  13:14:51 STEP 2939 H=2939:00 +CPU= 43.992
  13:16:16 STEP 2940 H=2940:00 +CPU= 78.691
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002940
  13:17:03 STEP 2941 H=2941:00 +CPU= 43.380
  13:17:49 STEP 2942 H=2942:00 +CPU= 43.157
  13:19:02 STEP 2943 H=2943:00 +CPU= 68.337
  13:19:51 STEP 2944 H=2944:00 +CPU= 45.017
  13:20:35 STEP 2945 H=2945:00 +CPU= 41.368
  13:21:50 STEP 2946 H=2946:00 +CPU= 69.127
  13:22:37 STEP 2947 H=2947:00 +CPU= 43.592
  13:23:23 STEP 2948 H=2948:00 +CPU= 43.044
  13:24:37 STEP 2949 H=2949:00 +CPU= 68.822
  13:25:27 STEP 2950 H=2950:00 +CPU= 46.275
  13:26:12 STEP 2951 H=2951:00 +CPU= 41.869
  13:27:35 STEP 2952 H=2952:00 +CPU= 77.720
The child process terminated with status: 0
Moving to projects directory: /var/lib/boinc/slots/13/ICMGGhq0f+002952
Moving to projects directory: /var/lib/boinc/slots/13/ICMSHhq0f+002952
Moving to projects directory: /var/lib/boinc/slots/13/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002940
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002952
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMGGhq0f+002940
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12202663/ICMSHhq0f+002940
Zipping up the final file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0019_2017050100_123_986_12202663_2_r1860330494_122.zip
Uploading the final file: upload_file_122.zip
Uploading trickle at timestep: 10623600
13:31:00 (9150): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:50:04 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 10,623,600 168,520 0.0159
09 May 2023 16:48:47 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 9,849,600 155,910 0.0158
09 May 2023 16:48:41 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 8,985,600 142,400 0.0158
09 May 2023 16:48:35 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 8,121,600 129,584 0.0160
09 May 2023 16:48:31 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 7,257,600 118,501 0.0163
09 May 2023 16:48:25 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 6,393,600 104,368 0.0163
09 May 2023 16:48:20 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 5,529,600 89,527 0.0162
09 May 2023 16:47:55 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 4,665,600 74,776 0.0160
09 May 2023 16:47:12 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 3,456,000 55,003 0.0159
09 May 2023 16:47:08 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 2,592,000 39,715 0.0153
09 May 2023 16:47:04 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 1,728,000 25,733 0.0149
09 May 2023 16:47:00 1510311 22305117 oifs_43r3_ps_0019_2017050100_123_986_12202663_2 864,000 12,844 0.0149


©2024 climateprediction.net