climateprediction.net home page
Task 22302295

Task 22302295

Name oifs_43r3_ps_0330_2013050100_123_982_12198974_2
Workunit 12198974
Created 23 Jan 2023, 9:32:30 UTC
Sent 23 Jan 2023, 9:54:26 UTC
Report deadline 24 Mar 2023, 9:54:26 UTC
Received 27 Jan 2023, 15:00:46 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 1 (0x00000001) Unknown error code
Computer ID 1536027
Run time 11 hours 35 min 57 sec
CPU time 11 hours 35 min 15 sec
Validate state Invalid
Credit 13,063.68
Device peak FLOPS 7.20 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,798.07 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,821.47 MB
Stderr
<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ion.net/oifs_43r3_ps_12198974/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_102.zip
Uploading the intermediate file: upload_file_102.zip
  20:23:17 STEP 2473 H=2473:00 +CPU= 12.728
  20:23:29 STEP 2474 H=2474:00 +CPU= 12.656
  20:23:47 STEP 2475 H=2475:00 +CPU= 17.696
  20:24:00 STEP 2476 H=2476:00 +CPU= 12.685
  20:24:11 STEP 2477 H=2477:00 +CPU= 11.662
  20:24:30 STEP 2478 H=2478:00 +CPU= 18.342
  20:24:42 STEP 2479 H=2479:00 +CPU= 12.068
  20:24:53 STEP 2480 H=2480:00 +CPU= 11.646
  20:25:11 STEP 2481 H=2481:00 +CPU= 17.359
  20:25:23 STEP 2482 H=2482:00 +CPU= 12.247
  20:25:35 STEP 2483 H=2483:00 +CPU= 11.880
  20:25:55 STEP 2484 H=2484:00 +CPU= 19.679
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002484
  20:26:06 STEP 2485 H=2485:00 +CPU= 11.742
  20:26:18 STEP 2486 H=2486:00 +CPU= 11.741
  20:26:37 STEP 2487 H=2487:00 +CPU= 18.479
  20:26:49 STEP 2488 H=2488:00 +CPU= 12.107
  20:27:01 STEP 2489 H=2489:00 +CPU= 11.794
  20:27:19 STEP 2490 H=2490:00 +CPU= 18.544
  20:27:31 STEP 2491 H=2491:00 +CPU= 11.664
  20:27:43 STEP 2492 H=2492:00 +CPU= 12.305
  20:28:01 STEP 2493 H=2493:00 +CPU= 18.165
  20:28:13 STEP 2494 H=2494:00 +CPU= 12.035
  20:28:26 STEP 2495 H=2495:00 +CPU= 12.198
  20:28:47 STEP 2496 H=2496:00 +CPU= 21.027
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_103.zip
Uploading the intermediate file: upload_file_103.zip
  20:28:58 STEP 2497 H=2497:00 +CPU= 10.997
  20:29:10 STEP 2498 H=2498:00 +CPU= 12.687
  20:29:29 STEP 2499 H=2499:00 +CPU= 19.021
  20:29:42 STEP 2500 H=2500:00 +CPU= 12.054
  20:29:53 STEP 2501 H=2501:00 +CPU= 11.179
  20:30:11 STEP 2502 H=2502:00 +CPU= 17.960
  20:30:22 STEP 2503 H=2503:00 +CPU= 11.485
  20:30:34 STEP 2504 H=2504:00 +CPU= 11.841
  20:30:52 STEP 2505 H=2505:00 +CPU= 17.603
  20:31:03 STEP 2506 H=2506:00 +CPU= 11.825
  20:31:15 STEP 2507 H=2507:00 +CPU= 11.653
  20:31:35 STEP 2508 H=2508:00 +CPU= 19.900
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002508
  20:31:47 STEP 2509 H=2509:00 +CPU= 11.931
  20:31:59 STEP 2510 H=2510:00 +CPU= 12.127
  20:32:17 STEP 2511 H=2511:00 +CPU= 18.132
  20:32:29 STEP 2512 H=2512:00 +CPU= 12.102
  20:32:42 STEP 2513 H=2513:00 +CPU= 12.356
  20:32:59 STEP 2514 H=2514:00 +CPU= 17.630
  20:33:11 STEP 2515 H=2515:00 +CPU= 11.278
  20:33:22 STEP 2516 H=2516:00 +CPU= 11.624
  20:33:39 STEP 2517 H=2517:00 +CPU= 16.932
  20:33:51 STEP 2518 H=2518:00 +CPU= 12.004
  20:34:03 STEP 2519 H=2519:00 +CPU= 11.428
  20:34:23 STEP 2520 H=2520:00 +CPU= 20.054
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_104.zip
Uploading the intermediate file: upload_file_104.zip
  20:34:34 STEP 2521 H=2521:00 +CPU= 11.020
  20:34:45 STEP 2522 H=2522:00 +CPU= 11.608
  20:35:04 STEP 2523 H=2523:00 +CPU= 18.321
  20:35:16 STEP 2524 H=2524:00 +CPU= 12.082
  20:35:27 STEP 2525 H=2525:00 +CPU= 11.527
  20:35:46 STEP 2526 H=2526:00 +CPU= 18.544
  20:35:57 STEP 2527 H=2527:00 +CPU= 11.291
  20:36:10 STEP 2528 H=2528:00 +CPU= 12.257
  20:36:27 STEP 2529 H=2529:00 +CPU= 17.348
  20:36:39 STEP 2530 H=2530:00 +CPU= 11.592
  20:36:50 STEP 2531 H=2531:00 +CPU= 11.700
  20:37:09 STEP 2532 H=2532:00 +CPU= 18.879
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002532
  20:37:20 STEP 2533 H=2533:00 +CPU= 11.209
  20:37:32 STEP 2534 H=2534:00 +CPU= 11.818
  20:37:49 STEP 2535 H=2535:00 +CPU= 17.003
  20:38:01 STEP 2536 H=2536:00 +CPU= 11.477
  20:38:12 STEP 2537 H=2537:00 +CPU= 11.374
  20:38:29 STEP 2538 H=2538:00 +CPU= 17.336
  20:38:41 STEP 2539 H=2539:00 +CPU= 11.640
  20:38:53 STEP 2540 H=2540:00 +CPU= 12.018
  20:39:10 STEP 2541 H=2541:00 +CPU= 17.262
  20:39:22 STEP 2542 H=2542:00 +CPU= 11.467
  20:39:34 STEP 2543 H=2543:00 +CPU= 11.924
  20:39:54 STEP 2544 H=2544:00 +CPU= 20.107
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_105.zip
Uploading the intermediate file: upload_file_105.zip
  20:40:06 STEP 2545 H=2545:00 +CPU= 12.386
  20:40:19 STEP 2546 H=2546:00 +CPU= 12.117
  20:40:36 STEP 2547 H=2547:00 +CPU= 17.400
  20:40:48 STEP 2548 H=2548:00 +CPU= 11.833
  20:40:59 STEP 2549 H=2549:00 +CPU= 11.501
  20:41:17 STEP 2550 H=2550:00 +CPU= 18.086
  20:41:28 STEP 2551 H=2551:00 +CPU= 11.087
  20:41:40 STEP 2552 H=2552:00 +CPU= 11.721
  20:41:57 STEP 2553 H=2553:00 +CPU= 17.176
  20:42:09 STEP 2554 H=2554:00 +CPU= 11.681
  20:42:20 STEP 2555 H=2555:00 +CPU= 11.254
  20:42:38 STEP 2556 H=2556:00 +CPU= 17.959
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002556
  20:42:50 STEP 2557 H=2557:00 +CPU= 11.283
  20:43:01 STEP 2558 H=2558:00 +CPU= 11.566
  20:43:20 STEP 2559 H=2559:00 +CPU= 18.393
  20:43:32 STEP 2560 H=2560:00 +CPU= 12.827
  20:43:44 STEP 2561 H=2561:00 +CPU= 11.742
  20:44:02 STEP 2562 H=2562:00 +CPU= 18.149
  20:44:14 STEP 2563 H=2563:00 +CPU= 11.703
  20:44:27 STEP 2564 H=2564:00 +CPU= 12.716
  20:44:44 STEP 2565 H=2565:00 +CPU= 17.590
  20:44:56 STEP 2566 H=2566:00 +CPU= 11.472
  20:45:07 STEP 2567 H=2567:00 +CPU= 11.560
  20:45:28 STEP 2568 H=2568:00 +CPU= 20.298
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002568
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_106.zip
Uploading the intermediate file: upload_file_106.zip
  20:45:39 STEP 2569 H=2569:00 +CPU= 11.290
  20:45:51 STEP 2570 H=2570:00 +CPU= 11.766
  20:46:08 STEP 2571 H=2571:00 +CPU= 17.184
  20:46:20 STEP 2572 H=2572:00 +CPU= 11.886
  20:46:32 STEP 2573 H=2573:00 +CPU= 11.589
  20:46:50 STEP 2574 H=2574:00 +CPU= 17.983
  20:47:01 STEP 2575 H=2575:00 +CPU= 11.432
  20:47:13 STEP 2576 H=2576:00 +CPU= 11.948
  20:47:30 STEP 2577 H=2577:00 +CPU= 17.213
  20:47:42 STEP 2578 H=2578:00 +CPU= 11.392
  20:47:53 STEP 2579 H=2579:00 +CPU= 11.545
  20:48:12 STEP 2580 H=2580:00 +CPU= 18.569
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002580
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002580
  20:48:23 STEP 2581 H=2581:00 +CPU= 11.305
  20:48:35 STEP 2582 H=2582:00 +CPU= 11.555
  20:48:52 STEP 2583 H=2583:00 +CPU= 17.439
  20:49:04 STEP 2584 H=2584:00 +CPU= 11.894
  20:49:16 STEP 2585 H=2585:00 +CPU= 11.568
  20:49:34 STEP 2586 H=2586:00 +CPU= 18.124
  20:49:45 STEP 2587 H=2587:00 +CPU= 11.034
  20:49:57 STEP 2588 H=2588:00 +CPU= 11.823
  20:50:14 STEP 2589 H=2589:00 +CPU= 17.498
  20:50:26 STEP 2590 H=2590:00 +CPU= 11.616
  20:50:38 STEP 2591 H=2591:00 +CPU= 12.600
  20:50:58 STEP 2592 H=2592:00 +CPU= 20.053
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002592
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_107.zip
Uploading the intermediate file: upload_file_107.zip
  20:51:10 STEP 2593 H=2593:00 +CPU= 11.264
  20:51:21 STEP 2594 H=2594:00 +CPU= 11.622
Uploading trickle at timestep: 9331200
  20:51:38 STEP 2595 H=2595:00 +CPU= 17.131
  20:51:50 STEP 2596 H=2596:00 +CPU= 11.406
  20:52:01 STEP 2597 H=2597:00 +CPU= 10.756
  20:52:18 STEP 2598 H=2598:00 +CPU= 17.472
  20:52:29 STEP 2599 H=2599:00 +CPU= 10.920
  20:52:40 STEP 2600 H=2600:00 +CPU= 11.458
  20:52:57 STEP 2601 H=2601:00 +CPU= 16.606
  20:53:09 STEP 2602 H=2602:00 +CPU= 12.134
  20:53:21 STEP 2603 H=2603:00 +CPU= 11.454
  20:53:39 STEP 2604 H=2604:00 +CPU= 18.614
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002604
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002604
  20:53:51 STEP 2605 H=2605:00 +CPU= 12.117
  20:54:03 STEP 2606 H=2606:00 +CPU= 12.037
  20:54:21 STEP 2607 H=2607:00 +CPU= 17.093
  20:54:32 STEP 2608 H=2608:00 +CPU= 11.575
  20:54:43 STEP 2609 H=2609:00 +CPU= 10.782
  20:55:01 STEP 2610 H=2610:00 +CPU= 17.826
  20:55:12 STEP 2611 H=2611:00 +CPU= 11.393
  20:55:24 STEP 2612 H=2612:00 +CPU= 11.885
  20:55:42 STEP 2613 H=2613:00 +CPU= 18.384
  20:55:54 STEP 2614 H=2614:00 +CPU= 11.599
  20:56:06 STEP 2615 H=2615:00 +CPU= 11.689
  20:56:26 STEP 2616 H=2616:00 +CPU= 20.087
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002616
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002604
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002604
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_108.zip
Uploading the intermediate file: upload_file_108.zip
  20:56:37 STEP 2617 H=2617:00 +CPU= 11.333
  20:56:49 STEP 2618 H=2618:00 +CPU= 11.815
  20:57:06 STEP 2619 H=2619:00 +CPU= 17.324
  20:57:19 STEP 2620 H=2620:00 +CPU= 12.076
  20:57:30 STEP 2621 H=2621:00 +CPU= 11.560
  20:57:48 STEP 2622 H=2622:00 +CPU= 17.591
  20:57:59 STEP 2623 H=2623:00 +CPU= 10.921
  20:58:10 STEP 2624 H=2624:00 +CPU= 11.455
  20:58:28 STEP 2625 H=2625:00 +CPU= 17.975
  20:58:40 STEP 2626 H=2626:00 +CPU= 11.766
  20:58:51 STEP 2627 H=2627:00 +CPU= 10.972
  20:59:10 STEP 2628 H=2628:00 +CPU= 18.795
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002628
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002628
  20:59:21 STEP 2629 H=2629:00 +CPU= 11.290
  20:59:33 STEP 2630 H=2630:00 +CPU= 11.881
  20:59:49 STEP 2631 H=2631:00 +CPU= 16.358
  21:00:01 STEP 2632 H=2632:00 +CPU= 12.207
  21:00:13 STEP 2633 H=2633:00 +CPU= 11.900
  21:00:31 STEP 2634 H=2634:00 +CPU= 18.108
  21:00:44 STEP 2635 H=2635:00 +CPU= 12.113
  21:00:56 STEP 2636 H=2636:00 +CPU= 12.037
  21:01:13 STEP 2637 H=2637:00 +CPU= 17.644
  21:01:25 STEP 2638 H=2638:00 +CPU= 12.096
  21:01:37 STEP 2639 H=2639:00 +CPU= 11.896
  21:01:58 STEP 2640 H=2640:00 +CPU= 20.543
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002640
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002616
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002628
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002628
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_109.zip
Uploading the intermediate file: upload_file_109.zip
  21:02:10 STEP 2641 H=2641:00 +CPU= 11.790
  21:02:22 STEP 2642 H=2642:00 +CPU= 11.947
  21:02:39 STEP 2643 H=2643:00 +CPU= 17.485
  21:02:52 STEP 2644 H=2644:00 +CPU= 12.469
  21:03:03 STEP 2645 H=2645:00 +CPU= 11.700
  21:03:21 STEP 2646 H=2646:00 +CPU= 17.976
  21:03:33 STEP 2647 H=2647:00 +CPU= 11.850
  21:03:45 STEP 2648 H=2648:00 +CPU= 12.003
  21:04:03 STEP 2649 H=2649:00 +CPU= 17.542
  21:04:15 STEP 2650 H=2650:00 +CPU= 11.802
  21:04:26 STEP 2651 H=2651:00 +CPU= 11.269
  21:04:45 STEP 2652 H=2652:00 +CPU= 18.976
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002652
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002652
  21:04:56 STEP 2653 H=2653:00 +CPU= 11.586
  21:05:09 STEP 2654 H=2654:00 +CPU= 12.178
  21:05:26 STEP 2655 H=2655:00 +CPU= 17.697
  21:05:39 STEP 2656 H=2656:00 +CPU= 12.823
  21:05:51 STEP 2657 H=2657:00 +CPU= 11.807
  21:06:08 STEP 2658 H=2658:00 +CPU= 16.890
  21:06:19 STEP 2659 H=2659:00 +CPU= 11.253
  21:06:31 STEP 2660 H=2660:00 +CPU= 11.669
  21:06:48 STEP 2661 H=2661:00 +CPU= 17.511
  21:07:00 STEP 2662 H=2662:00 +CPU= 11.482
  21:07:12 STEP 2663 H=2663:00 +CPU= 11.839
  21:07:32 STEP 2664 H=2664:00 +CPU= 20.310
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002664
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002640
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002652
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002652
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_110.zip
Uploading the intermediate file: upload_file_110.zip
  21:07:43 STEP 2665 H=2665:00 +CPU= 10.554
  21:07:54 STEP 2666 H=2666:00 +CPU= 11.298
  21:08:11 STEP 2667 H=2667:00 +CPU= 17.220
  21:08:23 STEP 2668 H=2668:00 +CPU= 12.051
  21:08:35 STEP 2669 H=2669:00 +CPU= 11.523
  21:08:53 STEP 2670 H=2670:00 +CPU= 18.196
  21:09:04 STEP 2671 H=2671:00 +CPU= 11.052
  21:09:15 STEP 2672 H=2672:00 +CPU= 11.223
  21:09:32 STEP 2673 H=2673:00 +CPU= 16.764
  21:09:43 STEP 2674 H=2674:00 +CPU= 11.360
  21:09:55 STEP 2675 H=2675:00 +CPU= 11.345
  21:10:13 STEP 2676 H=2676:00 +CPU= 18.769
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002676
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002676
  21:10:25 STEP 2677 H=2677:00 +CPU= 10.994
  21:10:37 STEP 2678 H=2678:00 +CPU= 12.449
  21:10:54 STEP 2679 H=2679:00 +CPU= 17.405
  21:11:07 STEP 2680 H=2680:00 +CPU= 12.016
  21:11:19 STEP 2681 H=2681:00 +CPU= 12.246
  21:11:37 STEP 2682 H=2682:00 +CPU= 17.746
  21:11:48 STEP 2683 H=2683:00 +CPU= 10.945
  21:12:00 STEP 2684 H=2684:00 +CPU= 11.974
  21:12:15 STEP 2685 H=2685:00 +CPU= 15.527
  21:12:25 STEP 2686 H=2686:00 +CPU= 10.197
  21:12:36 STEP 2687 H=2687:00 +CPU= 11.124
  21:12:54 STEP 2688 H=2688:00 +CPU= 18.036
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002688
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002664
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002676
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002676
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_111.zip
Uploading the intermediate file: upload_file_111.zip
  21:13:05 STEP 2689 H=2689:00 +CPU= 10.118
  21:13:15 STEP 2690 H=2690:00 +CPU= 10.619
  21:13:31 STEP 2691 H=2691:00 +CPU= 15.895
  21:13:42 STEP 2692 H=2692:00 +CPU= 10.922
  21:13:52 STEP 2693 H=2693:00 +CPU= 10.389
  21:14:08 STEP 2694 H=2694:00 +CPU= 16.022
  21:14:19 STEP 2695 H=2695:00 +CPU= 10.929
  21:14:30 STEP 2696 H=2696:00 +CPU= 11.039
  21:14:46 STEP 2697 H=2697:00 +CPU= 15.893
  21:14:56 STEP 2698 H=2698:00 +CPU=  9.986
  21:15:06 STEP 2699 H=2699:00 +CPU= 10.047
  21:15:25 STEP 2700 H=2700:00 +CPU= 18.685
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002700
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002700
  21:15:36 STEP 2701 H=2701:00 +CPU= 10.882
  21:15:48 STEP 2702 H=2702:00 +CPU= 12.134
  21:16:06 STEP 2703 H=2703:00 +CPU= 17.736
  21:16:18 STEP 2704 H=2704:00 +CPU= 11.800
  21:16:29 STEP 2705 H=2705:00 +CPU= 11.638
  21:16:48 STEP 2706 H=2706:00 +CPU= 18.525
  21:16:59 STEP 2707 H=2707:00 +CPU= 10.930
  21:17:11 STEP 2708 H=2708:00 +CPU= 11.827
  21:17:29 STEP 2709 H=2709:00 +CPU= 17.943
  21:17:40 STEP 2710 H=2710:00 +CPU= 11.454
  21:17:52 STEP 2711 H=2711:00 +CPU= 11.750
  21:18:13 STEP 2712 H=2712:00 +CPU= 20.908
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002712
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002688
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002700
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002700
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_112.zip
Uploading the intermediate file: upload_file_112.zip
  21:18:24 STEP 2713 H=2713:00 +CPU= 11.050
  21:18:36 STEP 2714 H=2714:00 +CPU= 12.406
  21:18:54 STEP 2715 H=2715:00 +CPU= 18.117
  21:19:06 STEP 2716 H=2716:00 +CPU= 11.154
  21:19:17 STEP 2717 H=2717:00 +CPU= 11.562
  21:19:35 STEP 2718 H=2718:00 +CPU= 17.987
  21:19:46 STEP 2719 H=2719:00 +CPU= 11.079
  21:19:58 STEP 2720 H=2720:00 +CPU= 11.595
  21:20:16 STEP 2721 H=2721:00 +CPU= 17.942
  21:20:27 STEP 2722 H=2722:00 +CPU= 11.093
  21:20:38 STEP 2723 H=2723:00 +CPU= 11.310
  21:20:58 STEP 2724 H=2724:00 +CPU= 19.397
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002724
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002724
  21:21:08 STEP 2725 H=2725:00 +CPU= 10.584
  21:21:20 STEP 2726 H=2726:00 +CPU= 11.796
  21:21:38 STEP 2727 H=2727:00 +CPU= 17.666
  21:21:50 STEP 2728 H=2728:00 +CPU= 12.362
  21:22:02 STEP 2729 H=2729:00 +CPU= 11.308
  21:22:19 STEP 2730 H=2730:00 +CPU= 17.687
  21:22:31 STEP 2731 H=2731:00 +CPU= 11.271
  21:22:42 STEP 2732 H=2732:00 +CPU= 11.719
  21:22:59 STEP 2733 H=2733:00 +CPU= 16.818
  21:23:10 STEP 2734 H=2734:00 +CPU= 11.380
  21:23:22 STEP 2735 H=2735:00 +CPU= 11.106
  21:23:42 STEP 2736 H=2736:00 +CPU= 20.432
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002736
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002712
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002724
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002724
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_113.zip
Uploading the intermediate file: upload_file_113.zip
  21:23:53 STEP 2737 H=2737:00 +CPU= 10.715
  21:24:04 STEP 2738 H=2738:00 +CPU= 11.630
  21:24:23 STEP 2739 H=2739:00 +CPU= 18.451
  21:24:34 STEP 2740 H=2740:00 +CPU= 11.220
  21:24:45 STEP 2741 H=2741:00 +CPU= 10.748
  21:25:03 STEP 2742 H=2742:00 +CPU= 17.800
  21:25:13 STEP 2743 H=2743:00 +CPU= 10.823
  21:25:25 STEP 2744 H=2744:00 +CPU= 11.553
  21:25:42 STEP 2745 H=2745:00 +CPU= 16.835
  21:25:53 STEP 2746 H=2746:00 +CPU= 11.072
  21:26:04 STEP 2747 H=2747:00 +CPU= 11.042
  21:26:23 STEP 2748 H=2748:00 +CPU= 18.833
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002748
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002748
  21:26:34 STEP 2749 H=2749:00 +CPU= 11.265
  21:26:46 STEP 2750 H=2750:00 +CPU= 11.548
  21:27:04 STEP 2751 H=2751:00 +CPU= 17.962
  21:27:16 STEP 2752 H=2752:00 +CPU= 12.071
  21:27:27 STEP 2753 H=2753:00 +CPU= 11.401
  21:27:45 STEP 2754 H=2754:00 +CPU= 17.986
  21:27:56 STEP 2755 H=2755:00 +CPU= 11.126
  21:28:08 STEP 2756 H=2756:00 +CPU= 11.482
  21:28:25 STEP 2757 H=2757:00 +CPU= 17.592
  21:28:37 STEP 2758 H=2758:00 +CPU= 11.676
  21:28:49 STEP 2759 H=2759:00 +CPU= 11.749
  21:29:09 STEP 2760 H=2760:00 +CPU= 20.279
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002760
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002736
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002748
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002748
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_114.zip
Uploading the intermediate file: upload_file_114.zip
  21:29:20 STEP 2761 H=2761:00 +CPU= 10.934
  21:29:32 STEP 2762 H=2762:00 +CPU= 11.415
  21:29:50 STEP 2763 H=2763:00 +CPU= 18.341
  21:30:01 STEP 2764 H=2764:00 +CPU= 11.337
  21:30:12 STEP 2765 H=2765:00 +CPU= 10.820
  21:30:31 STEP 2766 H=2766:00 +CPU= 18.306
  21:30:41 STEP 2767 H=2767:00 +CPU= 10.740
  21:30:53 STEP 2768 H=2768:00 +CPU= 11.216
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0330
(argv4) batchid: 982
(argv5) wuid: 12198974
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/2
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_fb36161694a2a8b0405ebe60f60816d0 to: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0330_2013050100_123_982_12198974.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/2/oifs_43r3_ps_0330_2013050100_123_982_12198974.zip
ic_ancil_file: ic_ancil_12198974
ifsdata_file: ifsdata_12198974
climate_data_file: clim_data_12198974
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_1a21b88218e8c65944b05af19fccddc2 to: /var/lib/boinc-client/slots/2/ic_ancil_12198974.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/2/ic_ancil_12198974.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_6cf0682eaad68f63fa8ae550090b29c0 to: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12198974.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/2/ifsdata/ifsdata_12198974.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/2/159l_2/clim_data_12198974.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/2/159l_2/clim_data_12198974.zip
Checking for progress XML file: /var/lib/boinc-client/slots/2/progress_file_12198974.xml
Opened progress file ok : /var/lib/boinc-client/slots/2/progress_file_12198974.xml
-- Model is restarting --
Adjusting last_iter, 2768, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/2/progress_file_12198974.xml
last_cpu_time: 38618
upload_file_number: 115
last_iter: 2761
last_upload: 9936000
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588
The child process has been launched with process id: 5138
Executing the command: /var/lib/boinc-client/slots/2/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:Biggles:1:1:5138:5138] [20230127:140007:1674828007:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  14:00:32 STEP 2760 H=2760:00 +CPU= 22.832
  14:00:44 STEP 2761 H=2761:00 +CPU= 11.740
  14:00:57 STEP 2762 H=2762:00 +CPU= 13.017
  14:01:16 STEP 2763 H=2763:00 +CPU= 19.554
  14:01:30 STEP 2764 H=2764:00 +CPU= 14.135
  14:01:46 STEP 2765 H=2765:00 +CPU= 14.512
  14:02:06 STEP 2766 H=2766:00 +CPU= 19.900
  14:02:20 STEP 2767 H=2767:00 +CPU= 14.011
  14:02:34 STEP 2768 H=2768:00 +CPU= 14.126
  14:02:54 STEP 2769 H=2769:00 +CPU= 19.832
  14:03:08 STEP 2770 H=2770:00 +CPU= 13.730
  14:03:21 STEP 2771 H=2771:00 +CPU= 13.790
  14:03:44 STEP 2772 H=2772:00 +CPU= 22.096
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002772
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002772
  14:03:57 STEP 2773 H=2773:00 +CPU= 13.109
  14:04:10 STEP 2774 H=2774:00 +CPU= 13.503
  14:04:30 STEP 2775 H=2775:00 +CPU= 19.505
  14:04:44 STEP 2776 H=2776:00 +CPU= 14.209
  14:04:57 STEP 2777 H=2777:00 +CPU= 13.508
  14:05:17 STEP 2778 H=2778:00 +CPU= 19.997
  14:05:31 STEP 2779 H=2779:00 +CPU= 13.925
  14:05:45 STEP 2780 H=2780:00 +CPU= 13.432
  14:06:04 STEP 2781 H=2781:00 +CPU= 19.547
  14:06:18 STEP 2782 H=2782:00 +CPU= 13.402
  14:06:32 STEP 2783 H=2783:00 +CPU= 13.901
  14:06:54 STEP 2784 H=2784:00 +CPU= 22.296
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002784
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002760
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002772
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002772
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_115.zip
Uploading the intermediate file: upload_file_115.zip
  14:07:07 STEP 2785 H=2785:00 +CPU= 13.049
  14:07:20 STEP 2786 H=2786:00 +CPU= 13.264
  14:07:39 STEP 2787 H=2787:00 +CPU= 18.950
  14:07:52 STEP 2788 H=2788:00 +CPU= 12.881
  14:08:04 STEP 2789 H=2789:00 +CPU= 12.061
  14:08:24 STEP 2790 H=2790:00 +CPU= 19.400
  14:08:36 STEP 2791 H=2791:00 +CPU= 12.304
  14:08:49 STEP 2792 H=2792:00 +CPU= 12.901
  14:09:09 STEP 2793 H=2793:00 +CPU= 20.276
  14:09:22 STEP 2794 H=2794:00 +CPU= 13.044
  14:09:36 STEP 2795 H=2795:00 +CPU= 13.274
  14:09:56 STEP 2796 H=2796:00 +CPU= 20.701
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002796
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002796
  14:10:09 STEP 2797 H=2797:00 +CPU= 12.583
  14:10:22 STEP 2798 H=2798:00 +CPU= 13.303
  14:10:41 STEP 2799 H=2799:00 +CPU= 18.874
  14:10:57 STEP 2800 H=2800:00 +CPU= 15.227
  14:11:10 STEP 2801 H=2801:00 +CPU= 13.648
  14:11:34 STEP 2802 H=2802:00 +CPU= 23.433
  14:11:48 STEP 2803 H=2803:00 +CPU= 14.165
  14:12:01 STEP 2804 H=2804:00 +CPU= 13.264
  14:12:22 STEP 2805 H=2805:00 +CPU= 20.350
  14:12:35 STEP 2806 H=2806:00 +CPU= 13.387
  14:12:48 STEP 2807 H=2807:00 +CPU= 13.250
  14:13:12 STEP 2808 H=2808:00 +CPU= 23.857
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002808
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002784
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002796
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002796
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_116.zip
Uploading the intermediate file: upload_file_116.zip
  14:13:26 STEP 2809 H=2809:00 +CPU= 13.708
  14:13:40 STEP 2810 H=2810:00 +CPU= 13.798
  14:14:00 STEP 2811 H=2811:00 +CPU= 20.671
  14:14:15 STEP 2812 H=2812:00 +CPU= 14.399
  14:14:28 STEP 2813 H=2813:00 +CPU= 13.300
  14:14:48 STEP 2814 H=2814:00 +CPU= 20.179
  14:15:02 STEP 2815 H=2815:00 +CPU= 14.052
  14:15:16 STEP 2816 H=2816:00 +CPU= 13.660
  14:15:36 STEP 2817 H=2817:00 +CPU= 19.595
  14:15:50 STEP 2818 H=2818:00 +CPU= 13.819
  14:16:03 STEP 2819 H=2819:00 +CPU= 13.002
  14:16:24 STEP 2820 H=2820:00 +CPU= 21.462
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002820
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002820
  14:16:38 STEP 2821 H=2821:00 +CPU= 13.649
  14:16:51 STEP 2822 H=2822:00 +CPU= 13.135
  14:17:11 STEP 2823 H=2823:00 +CPU= 20.146
  14:17:25 STEP 2824 H=2824:00 +CPU= 13.591
  14:17:38 STEP 2825 H=2825:00 +CPU= 12.892
  14:17:58 STEP 2826 H=2826:00 +CPU= 20.722
  14:18:12 STEP 2827 H=2827:00 +CPU= 13.336
  14:18:25 STEP 2828 H=2828:00 +CPU= 13.723
  14:18:45 STEP 2829 H=2829:00 +CPU= 19.713
  14:18:59 STEP 2830 H=2830:00 +CPU= 13.942
  14:19:13 STEP 2831 H=2831:00 +CPU= 13.758
  14:19:36 STEP 2832 H=2832:00 +CPU= 23.548
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002832
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002808
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002820
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002820
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_117.zip
Uploading the intermediate file: upload_file_117.zip
  14:19:50 STEP 2833 H=2833:00 +CPU= 13.558
  14:20:04 STEP 2834 H=2834:00 +CPU= 13.579
  14:20:25 STEP 2835 H=2835:00 +CPU= 21.150
  14:20:39 STEP 2836 H=2836:00 +CPU= 14.259
  14:20:52 STEP 2837 H=2837:00 +CPU= 13.030
  14:21:13 STEP 2838 H=2838:00 +CPU= 20.948
  14:21:27 STEP 2839 H=2839:00 +CPU= 13.965
  14:21:41 STEP 2840 H=2840:00 +CPU= 13.957
  14:22:01 STEP 2841 H=2841:00 +CPU= 19.677
  14:22:14 STEP 2842 H=2842:00 +CPU= 13.767
  14:22:28 STEP 2843 H=2843:00 +CPU= 13.823
  14:22:50 STEP 2844 H=2844:00 +CPU= 22.075
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002844
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002844
  14:23:04 STEP 2845 H=2845:00 +CPU= 13.901
  14:23:18 STEP 2846 H=2846:00 +CPU= 13.843
  14:23:39 STEP 2847 H=2847:00 +CPU= 20.254
  14:23:52 STEP 2848 H=2848:00 +CPU= 13.641
  14:24:06 STEP 2849 H=2849:00 +CPU= 13.790
  14:24:27 STEP 2850 H=2850:00 +CPU= 20.936
  14:24:40 STEP 2851 H=2851:00 +CPU= 13.288
  14:24:54 STEP 2852 H=2852:00 +CPU= 13.976
  14:25:15 STEP 2853 H=2853:00 +CPU= 20.553
  14:25:29 STEP 2854 H=2854:00 +CPU= 13.665
  14:25:42 STEP 2855 H=2855:00 +CPU= 13.540
  14:26:08 STEP 2856 H=2856:00 +CPU= 25.395
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002856
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002832
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002844
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002844
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_118.zip
Uploading the intermediate file: upload_file_118.zip
  14:26:22 STEP 2857 H=2857:00 +CPU= 14.199
  14:26:36 STEP 2858 H=2858:00 +CPU= 14.504
  14:26:57 STEP 2859 H=2859:00 +CPU= 20.717
  14:27:10 STEP 2860 H=2860:00 +CPU= 13.300
  14:27:23 STEP 2861 H=2861:00 +CPU= 13.020
  14:27:45 STEP 2862 H=2862:00 +CPU= 21.351
  14:27:58 STEP 2863 H=2863:00 +CPU= 13.208
  14:28:11 STEP 2864 H=2864:00 +CPU= 13.104
  14:28:32 STEP 2865 H=2865:00 +CPU= 20.385
  14:28:45 STEP 2866 H=2866:00 +CPU= 13.766
  14:28:58 STEP 2867 H=2867:00 +CPU= 12.907
  14:29:21 STEP 2868 H=2868:00 +CPU= 22.701
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002868
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002868
  14:29:34 STEP 2869 H=2869:00 +CPU= 13.244
  14:29:47 STEP 2870 H=2870:00 +CPU= 12.722
  14:30:07 STEP 2871 H=2871:00 +CPU= 19.947
  14:30:21 STEP 2872 H=2872:00 +CPU= 14.230
  14:30:35 STEP 2873 H=2873:00 +CPU= 13.299
  14:30:56 STEP 2874 H=2874:00 +CPU= 20.902
  14:31:09 STEP 2875 H=2875:00 +CPU= 13.113
  14:31:22 STEP 2876 H=2876:00 +CPU= 13.623
  14:31:43 STEP 2877 H=2877:00 +CPU= 20.575
  14:31:57 STEP 2878 H=2878:00 +CPU= 14.103
  14:32:11 STEP 2879 H=2879:00 +CPU= 13.416
  14:32:34 STEP 2880 H=2880:00 +CPU= 23.814
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002880
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002856
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002868
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002868
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_119.zip
Uploading the intermediate file: upload_file_119.zip
  14:32:48 STEP 2881 H=2881:00 +CPU= 13.156
  14:33:02 STEP 2882 H=2882:00 +CPU= 14.107
  14:33:22 STEP 2883 H=2883:00 +CPU= 20.621
  14:33:35 STEP 2884 H=2884:00 +CPU= 13.049
  14:33:48 STEP 2885 H=2885:00 +CPU= 12.856
  14:34:09 STEP 2886 H=2886:00 +CPU= 20.836
  14:34:22 STEP 2887 H=2887:00 +CPU= 12.989
  14:34:37 STEP 2888 H=2888:00 +CPU= 14.515
  14:34:57 STEP 2889 H=2889:00 +CPU= 19.785
  14:35:10 STEP 2890 H=2890:00 +CPU= 13.034
  14:35:23 STEP 2891 H=2891:00 +CPU= 13.112
  14:35:44 STEP 2892 H=2892:00 +CPU= 21.396
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002892
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002892
  14:35:58 STEP 2893 H=2893:00 +CPU= 13.599
  14:36:11 STEP 2894 H=2894:00 +CPU= 13.597
  14:36:31 STEP 2895 H=2895:00 +CPU= 19.642
  14:36:45 STEP 2896 H=2896:00 +CPU= 14.088
  14:36:58 STEP 2897 H=2897:00 +CPU= 12.925
  14:37:19 STEP 2898 H=2898:00 +CPU= 20.441
  14:37:32 STEP 2899 H=2899:00 +CPU= 13.479
  14:37:45 STEP 2900 H=2900:00 +CPU= 13.100
  14:38:05 STEP 2901 H=2901:00 +CPU= 20.030
  14:38:19 STEP 2902 H=2902:00 +CPU= 14.035
  14:38:33 STEP 2903 H=2903:00 +CPU= 13.591
  14:38:56 STEP 2904 H=2904:00 +CPU= 22.898
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002904
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002880
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002892
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002892
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_120.zip
Uploading the intermediate file: upload_file_120.zip
  14:39:10 STEP 2905 H=2905:00 +CPU= 14.240
  14:39:23 STEP 2906 H=2906:00 +CPU= 13.264
  14:39:44 STEP 2907 H=2907:00 +CPU= 20.674
  14:39:58 STEP 2908 H=2908:00 +CPU= 13.486
  14:40:11 STEP 2909 H=2909:00 +CPU= 12.861
  14:40:32 STEP 2910 H=2910:00 +CPU= 21.514
  14:40:46 STEP 2911 H=2911:00 +CPU= 13.909
  14:40:59 STEP 2912 H=2912:00 +CPU= 13.256
  14:41:20 STEP 2913 H=2913:00 +CPU= 21.014
  14:41:34 STEP 2914 H=2914:00 +CPU= 13.936
  14:41:47 STEP 2915 H=2915:00 +CPU= 12.819
  14:42:09 STEP 2916 H=2916:00 +CPU= 22.174
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002916
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002916
  14:42:22 STEP 2917 H=2917:00 +CPU= 13.096
  14:42:36 STEP 2918 H=2918:00 +CPU= 13.749
  14:42:56 STEP 2919 H=2919:00 +CPU= 20.135
  14:43:10 STEP 2920 H=2920:00 +CPU= 13.186
  14:43:23 STEP 2921 H=2921:00 +CPU= 13.219
  14:43:44 STEP 2922 H=2922:00 +CPU= 20.953
  14:43:57 STEP 2923 H=2923:00 +CPU= 12.820
  14:44:10 STEP 2924 H=2924:00 +CPU= 13.368
  14:44:30 STEP 2925 H=2925:00 +CPU= 20.297
  14:44:44 STEP 2926 H=2926:00 +CPU= 13.174
  14:44:57 STEP 2927 H=2927:00 +CPU= 13.903
  14:45:21 STEP 2928 H=2928:00 +CPU= 23.384
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002928
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002928
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002904
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMGGhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMSHhq0f+002916
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12198974/ICMUAhq0f+002916
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0330_2013050100_123_982_12198974_2_r1821042588_121.zip
Uploading the intermediate file: upload_file_121.zip
  14:45:35 STEP 2929 H=2929:00 +CPU= 13.902
  14:45:49 STEP 2930 H=2930:00 +CPU= 13.842
  14:46:09 STEP 2931 H=2931:00 +CPU= 20.564
  14:46:23 STEP 2932 H=2932:00 +CPU= 13.699
  14:46:36 STEP 2933 H=2933:00 +CPU= 13.551
  14:46:58 STEP 2934 H=2934:00 +CPU= 21.190
  14:47:11 STEP 2935 H=2935:00 +CPU= 13.447
  14:47:24 STEP 2936 H=2936:00 +CPU= 12.929
  14:47:45 STEP 2937 H=2937:00 +CPU= 21.253
  14:47:59 STEP 2938 H=2938:00 +CPU= 13.757
  14:48:12 STEP 2939 H=2939:00 +CPU= 13.136
  14:48:36 STEP 2940 H=2940:00 +CPU= 23.526
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMGGhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMSHhq0f+002940
Moving to projects directory: /var/lib/boinc-client/slots/2/ICMUAhq0f+002940
  14:48:49 STEP 2941 H=2941:00 +CPU= 13.224
  14:49:03 STEP 2942 H=2942:00 +CPU= 13.746
  14:49:25 STEP 2943 H=2943:00 +CPU= 20.429
  14:49:38 STEP 2944 H=2944:00 +CPU= 13.282
  14:49:52 STEP 2945 H=2945:00 +CPU= 13.978
  14:50:13 STEP 2946 H=2946:00 +CPU= 20.987
  14:50:27 STEP 2947 H=2947:00 +CPU= 13.851
  14:50:41 STEP 2948 H=2948:00 +CPU= 14.047
  14:51:01 STEP 2949 H=2949:00 +CPU= 20.058
  14:51:14 STEP 2950 H=2950:00 +CPU= 13.540
  14:51:28 STEP 2951 H=2951:00 +CPU= 13.786
  14:51:51 STEP 2952 H=2952:00 +CPU= 23.131
..The child process terminated with status: 0
>>> Printing last 70 lines from file: NODE.001_01


   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20130831234500  !!!!!!!!!!!! 

   !!!!!!!!!!!!!! WAVE FIELDS INTEGRATED.  DATE IS: 20130901000000  !!!!!!!!!!!! 
  
  GLOBAL NORM OF FIELDS RETURNED TO IFS :
  Charnock     1.335173974164972E-002  7.500100765334979E-003
  7.784413002369898E-002       26160
              HEX: 3F8B58282C47DCC0  3F7EB86CF80FA3C8  3FB3ED97C8A329F3
  U-Stokes     6.824393396186942E-003 -0.225402899793270     
  0.290997219991027            26160
              HEX: 3F7BF3E5273A0130  BFCCDA0091849074  3FD29FB2CDC5A7F8
  V-Stokes     2.664022833701577E-003 -0.342103076684375     
  0.267426524939980            26160
              HEX: 3F65D2DC5E4A19D4  BFD5E5044D8E1A87  3FD11D8424ACCF25
  
  
 END OF WAVE MODEL CALL 
 ----------------------------------------------------------------------- 
 14:51:51 STEP 2952 H=2952:00 +CPU= 23.131
 END CNT3
 NSTEP =  2952 CNT0   000000000  20.136  20.136  20.172   0.998               0               0               0               0               0
IO-STREAM STATISTICS, TOTAL NO OF REC/BYTES READ           35/   804992476 WRITTEN         2648/************
 -TASK-OPENED-OPEN-RECS IN -KBYTE IN    -RECS OUT-KBYTE OUT-WALL   -WALL IN-WALL
  OU-TOT IN -TOT OUT
   1    107    0       35       786125     2648 ************     7.7     0.8     6.9     1.1    10.8
===-=== START OF TIMING STATISTICS ===-===
 
STATS FOR ALL TASKS
 NUM ROUTINE                                     CALLS  MEAN(ms)   MAX(ms)   FRAC(%)  UNBAL(%)
   0 CNT0     - COMPLETE EXECUTION                   1 3104189.5 3104189.5    100.00      0.00
   1 CNT4     - FORWARD INTEGRATION                  1 3102533.1 3102533.1     99.95      0.00
   8 SCAN2M - GRID-POINT DYNAMICS                  209    1361.1    1361.1      9.16      0.00
   9 SPCM     - SPECTRAL COMP.                     192     238.1     238.1      1.47      0.00
  10 SCAN2M - PHYSICS                              193    6864.6    6864.6     42.68      0.00
  11 IOPACK   - OUTPUT P.P. RESULTS                 16     620.8     620.8      0.32      0.00
  12 SPNORM   - SPECTRAL NORM COMP.                193      11.9      11.9      0.07      0.00
  13 SCAN2M - RADIATION CALC.                       65    6877.4    6877.4     14.40      0.00
  17 GRIDFPOS IN CNT4                               16      10.8      10.8      0.01      0.00
  27 RESTART FILES - WRITING                         8    1241.6    1241.6      0.32      0.00
  28 RESTART FILES - READING                         1    1441.1    1441.1      0.05      0.00
  29 SU4FPOS IN CNT4                                16       0.1       0.1      0.00      0.00
  30 DYNFPOS IN CNT4                                16    1579.0    1579.0      0.81      0.00
  31 POSDDH IN STEPO                                 1       0.8       0.8      0.00      0.00
  37 CPGLAG   - SL COMPUTATIONS                    193    2208.8    2208.8     13.73      0.00
  38 WAM      - TOTAL COST OF WAVE MODEL           193    1670.5    1670.5     10.39      0.00
  39 SU0YOMB                                         1     148.3     148.3      0.00      0.00
  51 SCAN2M   - SL COMM. PART 1                    193      21.2      21.2      0.13      0.00
  54 SPCM     - M TO S/S TO M TRANSP.              192     213.8     213.8      1.32      0.00
  55 SPCIMPF  - S TO M/M TO S TRANSP.              192      34.6      34.6      0.21      0.00
  56 SPNORM   - SPECTRAL NORM COMM.                193       0.1       0.1      0.00      0.00
 102 LTINV_CTL   - INVERSE LEGENDRE TRANSFORM      662      81.7      81.7      1.74      0.00
 103 LTDIR_CTL   - DIRECT LEGENDRE TRANSFORM       416      76.7      76.7      1.03      0.00
 106 FTDIR_CTL   - DIRECT FOURIER TRANSFORM        416      30.0      30.0      0.40      0.00
 107 FTINV_CTL   - INVERSE FOURIER TRANSFORM       662      33.2      33.2      0.71      0.00
 140 SULEG       - COMP. OF LEGENDRE POL.            2      10.1      10.1      0.00      0.00
 152 LTINV_CTL   - M TO L TRANSPOSITION            662      11.8      11.8      0.25      0.00
 153 LTDIR_CTL   - L TO M TRANSPOSITION            416      10.3      10.3      0.14      0.00
 157 FTINV_CTL   - L TO G TRANSPOSITION            662      15.1      15.1      0.32      0.00
 158 FTDIR_CTL   - G TO L TRANSPOSITION            416      10.0      10.0      0.13      0.00
 400 GSTATS                                      46761       0.0       0.0      0.00      0.00
 401 GSTATS HOOK                                 44846       0.0       0.0      0.00      0.00
TOTAL MEASURED IMBALANCE =       0.0 SECONDS,  0.0 PERCENT
TOTAL WALLCLOCK TIME     3104.2 CPU TIME    3097.2 VECTOR TIME     3097.2


===-=== END   OF TIMING STATISTICS ===-===
FORECAST DAYS PER DAY     3425.3
  *** END CNT0 *** 
------------------------------------------------
>>> Printing last 11 lines from file: rcf

 CSTEP   = "    2952",
 CTIME   = "01230000      ",
 NSTEPLPP        =         201,         202,          49,      228226,      228227, 5*0,
 IPRGPNSRES      =           1,
 IPRGPEWRES      =           1,
 IPRTRWRES       =           1,
 IPRTRVRES       =           1,
 GMASS0  =   98315.9569433515     ,
 GMASSI  =   98315.9670191047     
 /
------------------------------------------------
>>> Printing last 17 lines from file: waminfo

PARAM=10U/10V,                                                          
REPRES=GG,                                                              
ANALYSIS PERIOD =       0                                               
FORECAST PERIOD =    9936000                                            
WIND TIME STEP IN SECONDS =    3600                                     
ANALYSIS FROM 20130501000000 TO 20130501000000                          
ASSIMILATION NO                                                         
NEW FORECAST NO                                                         
STATUS STORM1= UNFINISHED                                               
STATUS STORM2= UNFINISHED                                               
STATUS STORM3= UNFINISHED                                               
DATE FOR OUTPUT OF BOTH RESTART FILES = 00000000000000                  
LAST DATE FOR SPECTRA FILE OUTPUT = 20130901000000                      
BEGIN DATE FOR USING SURFACE CURRENT = 20130501000000                   
PARALLEL RESTART = YES                                                  
NUMBER OF MPI TASKS USED =          1                                   
------------------------------------------------
>>> Printing last 8 lines from file: /var/lib/boinc-client/slots/2/progress_file_12198974.xml

<running_values>
  <last_cpu_time>41714.920000</last_cpu_time>
  <upload_file_number>122</upload_file_number>
  <last_iter>2952</last_iter>
  <last_upload>10540800</last_upload>
  <model_completed>0</model_completed>
</running_values>
------------------------------------------------
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:42:30 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 9,331,200 36,242 0.0039
09 May 2023 16:42:26 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 8,467,200 32,874 0.0039
09 May 2023 16:42:23 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 7,603,200 29,536 0.0039
09 May 2023 16:42:21 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 6,739,200 26,228 0.0039
09 May 2023 16:42:18 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 5,875,200 22,911 0.0039
09 May 2023 16:42:16 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 5,011,200 19,475 0.0039
09 May 2023 16:42:16 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 4,147,200 16,118 0.0039
09 May 2023 16:42:12 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 3,283,200 12,772 0.0039
09 May 2023 16:42:10 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 2,419,200 9,466 0.0039
09 May 2023 16:42:01 1536027 22302295 oifs_43r3_ps_0330_2013050100_123_982_12198974_2 864,000 3,315 0.0038


©2024 climateprediction.net