climateprediction.net home page
Task 22265277

Task 22265277

Name oifs_43r3_ps_0674_1994050100_123_963_12180318_0
Workunit 12180318
Created 22 Dec 2022, 6:21:22 UTC
Sent 21 Jan 2023, 8:48:33 UTC
Report deadline 22 Mar 2023, 8:48:33 UTC
Received 25 Jan 2023, 20:53:48 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1410588
Run time 14 hours 56 min 56 sec
CPU time 14 hours 54 min 11 sec
Validate state Invalid
Credit 13,184.64
Device peak FLOPS 6.86 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,415.97 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,755.04 MB
Stderr
<core_client_version>7.16.14</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
ateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002400
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002412
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002412
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002412
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_100.zip
Uploading the intermediate file: upload_file_100.zip
  18:18:34 STEP 2425 H=2425:00 +CPU= 11.304
  18:18:46 STEP 2426 H=2426:00 +CPU= 11.556
  18:19:04 STEP 2427 H=2427:00 +CPU= 17.782
  18:19:15 STEP 2428 H=2428:00 +CPU= 11.602
  18:19:26 STEP 2429 H=2429:00 +CPU= 11.187
  18:19:45 STEP 2430 H=2430:00 +CPU= 18.357
  18:19:56 STEP 2431 H=2431:00 +CPU= 11.360
  18:20:08 STEP 2432 H=2432:00 +CPU= 11.452
  18:20:26 STEP 2433 H=2433:00 +CPU= 17.966
  18:20:37 STEP 2434 H=2434:00 +CPU= 11.537
  18:20:48 STEP 2435 H=2435:00 +CPU= 11.147
  18:21:08 STEP 2436 H=2436:00 +CPU= 19.923
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002436
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002436
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002436
  18:21:19 STEP 2437 H=2437:00 +CPU= 11.067
  18:21:31 STEP 2438 H=2438:00 +CPU= 11.381
  18:21:48 STEP 2439 H=2439:00 +CPU= 17.860
  18:22:00 STEP 2440 H=2440:00 +CPU= 11.577
  18:22:11 STEP 2441 H=2441:00 +CPU= 11.187
  18:22:30 STEP 2442 H=2442:00 +CPU= 18.397
  18:22:41 STEP 2443 H=2443:00 +CPU= 11.283
  18:22:52 STEP 2444 H=2444:00 +CPU= 11.547
  18:23:10 STEP 2445 H=2445:00 +CPU= 17.919
  18:23:22 STEP 2446 H=2446:00 +CPU= 11.661
  18:23:33 STEP 2447 H=2447:00 +CPU= 11.410
  18:23:55 STEP 2448 H=2448:00 +CPU= 21.287
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002448
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002448
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002448
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002424
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002424
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002424
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002436
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002436
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002436
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_101.zip
Uploading the intermediate file: upload_file_101.zip
  18:24:06 STEP 2449 H=2449:00 +CPU= 11.101
  18:24:17 STEP 2450 H=2450:00 +CPU= 11.588
  18:24:41 STEP 2451 H=2451:00 +CPU= 23.216
  18:24:57 STEP 2452 H=2452:00 +CPU= 16.030
  18:25:12 STEP 2453 H=2453:00 +CPU= 15.549
  18:25:37 STEP 2454 H=2454:00 +CPU= 24.921
  18:25:53 STEP 2455 H=2455:00 +CPU= 15.399
  18:26:09 STEP 2456 H=2456:00 +CPU= 16.095
  18:26:30 STEP 2457 H=2457:00 +CPU= 20.974
  18:26:41 STEP 2458 H=2458:00 +CPU= 11.634
  18:26:52 STEP 2459 H=2459:00 +CPU= 11.060
  18:27:19 STEP 2460 H=2460:00 +CPU= 27.009
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002460
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002460
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002460
  18:27:35 STEP 2461 H=2461:00 +CPU= 15.913
  18:27:47 STEP 2462 H=2462:00 +CPU= 11.501
  18:28:10 STEP 2463 H=2463:00 +CPU= 22.859
  18:28:24 STEP 2464 H=2464:00 +CPU= 14.158
  18:28:35 STEP 2465 H=2465:00 +CPU= 11.312
  18:28:54 STEP 2466 H=2466:00 +CPU= 18.460
  18:29:05 STEP 2467 H=2467:00 +CPU= 11.360
  18:29:17 STEP 2468 H=2468:00 +CPU= 11.690
  18:29:35 STEP 2469 H=2469:00 +CPU= 18.551
  18:29:46 STEP 2470 H=2470:00 +CPU= 11.364
  18:29:58 STEP 2471 H=2471:00 +CPU= 11.685
  18:30:25 STEP 2472 H=2472:00 +CPU= 26.795
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002472
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002472
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002472
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002448
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002448
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002448
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002460
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002460
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002460
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_102.zip
Uploading the intermediate file: upload_file_102.zip
  18:30:41 STEP 2473 H=2473:00 +CPU= 15.823
  18:30:57 STEP 2474 H=2474:00 +CPU= 16.255
  18:31:19 STEP 2475 H=2475:00 +CPU= 21.897
  18:31:31 STEP 2476 H=2476:00 +CPU= 11.789
  18:31:42 STEP 2477 H=2477:00 +CPU= 11.368
  18:32:01 STEP 2478 H=2478:00 +CPU= 18.706
  18:32:12 STEP 2479 H=2479:00 +CPU= 11.609
  18:32:24 STEP 2480 H=2480:00 +CPU= 11.618
  18:32:42 STEP 2481 H=2481:00 +CPU= 18.375
  18:32:58 STEP 2482 H=2482:00 +CPU= 15.324
  18:33:14 STEP 2483 H=2483:00 +CPU= 15.872
  18:33:42 STEP 2484 H=2484:00 +CPU= 27.983
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002484
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002484
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002484
  18:33:58 STEP 2485 H=2485:00 +CPU= 16.026
  18:34:14 STEP 2486 H=2486:00 +CPU= 16.338
  18:34:39 STEP 2487 H=2487:00 +CPU= 25.327
  18:34:56 STEP 2488 H=2488:00 +CPU= 16.406
  18:35:11 STEP 2489 H=2489:00 +CPU= 15.654
  18:35:37 STEP 2490 H=2490:00 +CPU= 25.902
  18:35:54 STEP 2491 H=2491:00 +CPU= 16.211
  18:36:10 STEP 2492 H=2492:00 +CPU= 16.190
  18:36:35 STEP 2493 H=2493:00 +CPU= 25.100
  18:36:51 STEP 2494 H=2494:00 +CPU= 16.487
  18:37:08 STEP 2495 H=2495:00 +CPU= 16.453
  18:37:37 STEP 2496 H=2496:00 +CPU= 29.611
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002496
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002496
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002496
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002472
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002472
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002472
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002484
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002484
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002484
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_103.zip
Uploading the intermediate file: upload_file_103.zip
  18:37:53 STEP 2497 H=2497:00 +CPU= 15.835
  18:38:10 STEP 2498 H=2498:00 +CPU= 16.522
  18:38:35 STEP 2499 H=2499:00 +CPU= 25.542
  18:38:52 STEP 2500 H=2500:00 +CPU= 16.383
  18:39:08 STEP 2501 H=2501:00 +CPU= 15.952
  18:39:34 STEP 2502 H=2502:00 +CPU= 25.932
  18:39:50 STEP 2503 H=2503:00 +CPU= 16.025
  18:40:06 STEP 2504 H=2504:00 +CPU= 16.311
  18:40:31 STEP 2505 H=2505:00 +CPU= 25.476
  18:40:48 STEP 2506 H=2506:00 +CPU= 16.426
  18:41:04 STEP 2507 H=2507:00 +CPU= 15.851
  18:41:32 STEP 2508 H=2508:00 +CPU= 28.096
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002508
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002508
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002508
  18:41:47 STEP 2509 H=2509:00 +CPU= 15.545
  18:42:04 STEP 2510 H=2510:00 +CPU= 16.316
  18:42:29 STEP 2511 H=2511:00 +CPU= 25.342
  18:42:46 STEP 2512 H=2512:00 +CPU= 16.487
  18:43:01 STEP 2513 H=2513:00 +CPU= 15.757
  18:43:27 STEP 2514 H=2514:00 +CPU= 26.124
  18:43:43 STEP 2515 H=2515:00 +CPU= 15.533
  18:43:59 STEP 2516 H=2516:00 +CPU= 16.336
  18:44:24 STEP 2517 H=2517:00 +CPU= 24.847
  18:44:41 STEP 2518 H=2518:00 +CPU= 16.425
  18:44:57 STEP 2519 H=2519:00 +CPU= 16.267
  18:45:26 STEP 2520 H=2520:00 +CPU= 29.378
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002520
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002520
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002520
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002496
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002496
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002496
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002508
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002508
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002508
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_104.zip
Uploading the intermediate file: upload_file_104.zip
  18:45:42 STEP 2521 H=2521:00 +CPU= 15.509
  18:45:58 STEP 2522 H=2522:00 +CPU= 16.062
  18:46:23 STEP 2523 H=2523:00 +CPU= 25.270
  18:46:39 STEP 2524 H=2524:00 +CPU= 16.197
  18:46:55 STEP 2525 H=2525:00 +CPU= 15.660
  18:47:21 STEP 2526 H=2526:00 +CPU= 25.966
  18:47:37 STEP 2527 H=2527:00 +CPU= 15.964
  18:47:53 STEP 2528 H=2528:00 +CPU= 16.434
  18:48:19 STEP 2529 H=2529:00 +CPU= 25.174
  18:48:35 STEP 2530 H=2530:00 +CPU= 16.110
  18:48:50 STEP 2531 H=2531:00 +CPU= 15.769
  18:49:18 STEP 2532 H=2532:00 +CPU= 28.010
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002532
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002532
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002532
  18:49:34 STEP 2533 H=2533:00 +CPU= 15.763
  18:49:51 STEP 2534 H=2534:00 +CPU= 16.452
  18:50:16 STEP 2535 H=2535:00 +CPU= 25.255
  18:50:32 STEP 2536 H=2536:00 +CPU= 16.099
  18:50:48 STEP 2537 H=2537:00 +CPU= 15.665
  18:51:13 STEP 2538 H=2538:00 +CPU= 25.648
  18:51:29 STEP 2539 H=2539:00 +CPU= 15.821
  18:51:46 STEP 2540 H=2540:00 +CPU= 16.435
  18:52:11 STEP 2541 H=2541:00 +CPU= 24.955
  18:52:27 STEP 2542 H=2542:00 +CPU= 16.189
  18:52:42 STEP 2543 H=2543:00 +CPU= 15.371
  18:53:10 STEP 2544 H=2544:00 +CPU= 28.019
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002544
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002544
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002544
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002520
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002520
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002520
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002532
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002532
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002532
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:53:25 STEP 2545 H=2545:00 +CPU= 15.038
  18:53:41 STEP 2546 H=2546:00 +CPU= 15.336
  18:54:05 STEP 2547 H=2547:00 +CPU= 24.018
  18:54:20 STEP 2548 H=2548:00 +CPU= 15.375
  18:54:35 STEP 2549 H=2549:00 +CPU= 15.053
  18:55:00 STEP 2550 H=2550:00 +CPU= 24.614
  18:55:15 STEP 2551 H=2551:00 +CPU= 15.347
  18:55:31 STEP 2552 H=2552:00 +CPU= 15.562
  18:55:54 STEP 2553 H=2553:00 +CPU= 23.796
  18:56:10 STEP 2554 H=2554:00 +CPU= 15.507
  18:56:25 STEP 2555 H=2555:00 +CPU= 15.141
  18:56:52 STEP 2556 H=2556:00 +CPU= 26.882
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002556
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002556
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002556
  18:57:07 STEP 2557 H=2557:00 +CPU= 15.088
  18:57:23 STEP 2558 H=2558:00 +CPU= 15.555
  18:57:46 STEP 2559 H=2559:00 +CPU= 23.815
  18:58:02 STEP 2560 H=2560:00 +CPU= 15.574
  18:58:17 STEP 2561 H=2561:00 +CPU= 15.074
  18:58:42 STEP 2562 H=2562:00 +CPU= 24.645
  18:58:57 STEP 2563 H=2563:00 +CPU= 15.051
  18:59:12 STEP 2564 H=2564:00 +CPU= 15.489
  18:59:36 STEP 2565 H=2565:00 +CPU= 23.759
  18:59:51 STEP 2566 H=2566:00 +CPU= 15.504
  19:00:07 STEP 2567 H=2567:00 +CPU= 15.476
  19:00:35 STEP 2568 H=2568:00 +CPU= 28.194
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002568
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002568
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002568
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002544
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002544
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002544
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002556
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002556
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002556
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_106.zip
Uploading the intermediate file: upload_file_106.zip
  19:00:50 STEP 2569 H=2569:00 +CPU= 15.111
  19:01:06 STEP 2570 H=2570:00 +CPU= 15.478
  19:01:29 STEP 2571 H=2571:00 +CPU= 23.700
  19:01:45 STEP 2572 H=2572:00 +CPU= 15.583
  19:02:00 STEP 2573 H=2573:00 +CPU= 15.010
  19:02:25 STEP 2574 H=2574:00 +CPU= 24.480
  19:02:39 STEP 2575 H=2575:00 +CPU= 14.899
  19:02:55 STEP 2576 H=2576:00 +CPU= 15.659
  19:03:19 STEP 2577 H=2577:00 +CPU= 24.160
  19:03:35 STEP 2578 H=2578:00 +CPU= 15.654
  19:03:50 STEP 2579 H=2579:00 +CPU= 15.163
  19:04:17 STEP 2580 H=2580:00 +CPU= 27.146
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002580
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002580
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002580
  19:04:32 STEP 2581 H=2581:00 +CPU= 15.032
  19:04:48 STEP 2582 H=2582:00 +CPU= 16.217
  19:05:13 STEP 2583 H=2583:00 +CPU= 24.059
  19:05:28 STEP 2584 H=2584:00 +CPU= 15.702
  19:05:44 STEP 2585 H=2585:00 +CPU= 15.358
  19:06:08 STEP 2586 H=2586:00 +CPU= 24.709
  19:06:23 STEP 2587 H=2587:00 +CPU= 15.168
  19:06:39 STEP 2588 H=2588:00 +CPU= 15.699
  19:07:03 STEP 2589 H=2589:00 +CPU= 24.167
  19:07:19 STEP 2590 H=2590:00 +CPU= 15.561
  19:07:35 STEP 2591 H=2591:00 +CPU= 15.825
  19:08:03 STEP 2592 H=2592:00 +CPU= 28.787
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002592
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002592
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002592
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002568
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002568
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002568
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002580
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002580
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002580
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_107.zip
Uploading the intermediate file: upload_file_107.zip
  19:08:19 STEP 2593 H=2593:00 +CPU= 15.100
  19:08:34 STEP 2594 H=2594:00 +CPU= 15.878
  19:08:59 STEP 2595 H=2595:00 +CPU= 24.288
  19:09:14 STEP 2596 H=2596:00 +CPU= 15.734
  19:09:30 STEP 2597 H=2597:00 +CPU= 15.067
  19:09:54 STEP 2598 H=2598:00 +CPU= 24.677
  19:10:10 STEP 2599 H=2599:00 +CPU= 15.394
  19:10:26 STEP 2600 H=2600:00 +CPU= 16.026
  19:10:50 STEP 2601 H=2601:00 +CPU= 24.171
  19:11:06 STEP 2602 H=2602:00 +CPU= 15.699
  19:11:21 STEP 2603 H=2603:00 +CPU= 15.257
  19:11:48 STEP 2604 H=2604:00 +CPU= 27.260
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002604
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002604
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002604
  19:12:03 STEP 2605 H=2605:00 +CPU= 15.130
  19:12:19 STEP 2606 H=2606:00 +CPU= 15.743
  19:12:39 STEP 2607 H=2607:00 +CPU= 19.932
  19:12:50 STEP 2608 H=2608:00 +CPU= 11.202
  19:13:01 STEP 2609 H=2609:00 +CPU= 11.108
  19:13:19 STEP 2610 H=2610:00 +CPU= 17.801
  19:13:30 STEP 2611 H=2611:00 +CPU= 10.873
  19:13:46 STEP 2612 H=2612:00 +CPU= 15.559
  19:14:10 STEP 2613 H=2613:00 +CPU= 24.138
  19:14:25 STEP 2614 H=2614:00 +CPU= 15.653
  19:14:41 STEP 2615 H=2615:00 +CPU= 15.796
  19:15:10 STEP 2616 H=2616:00 +CPU= 28.610
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002616
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002616
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002616
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002592
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002592
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002592
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002604
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002604
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002604
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_108.zip
Uploading the intermediate file: upload_file_108.zip
  19:15:25 STEP 2617 H=2617:00 +CPU= 15.312
Uploading trickle at timestep: 9417600
  19:15:41 STEP 2618 H=2618:00 +CPU= 15.954
  19:16:05 STEP 2619 H=2619:00 +CPU= 24.225
  19:16:21 STEP 2620 H=2620:00 +CPU= 15.680
  19:16:36 STEP 2621 H=2621:00 +CPU= 15.047
  19:17:01 STEP 2622 H=2622:00 +CPU= 25.056
  19:17:16 STEP 2623 H=2623:00 +CPU= 15.153
  19:17:32 STEP 2624 H=2624:00 +CPU= 15.747
  19:17:56 STEP 2625 H=2625:00 +CPU= 24.288
  19:18:12 STEP 2626 H=2626:00 +CPU= 15.721
  19:18:27 STEP 2627 H=2627:00 +CPU= 15.487
  19:18:56 STEP 2628 H=2628:00 +CPU= 28.052
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002628
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002628
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002628
  19:19:11 STEP 2629 H=2629:00 +CPU= 15.618
  19:19:27 STEP 2630 H=2630:00 +CPU= 16.200
  19:19:52 STEP 2631 H=2631:00 +CPU= 24.886
  19:20:08 STEP 2632 H=2632:00 +CPU= 16.050
  19:20:24 STEP 2633 H=2633:00 +CPU= 15.658
  19:20:49 STEP 2634 H=2634:00 +CPU= 25.079
  19:21:04 STEP 2635 H=2635:00 +CPU= 15.270
  19:21:20 STEP 2636 H=2636:00 +CPU= 15.844
  19:21:44 STEP 2637 H=2637:00 +CPU= 24.132
  19:22:00 STEP 2638 H=2638:00 +CPU= 15.688
  19:22:16 STEP 2639 H=2639:00 +CPU= 15.777
  19:22:44 STEP 2640 H=2640:00 +CPU= 28.607
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002640
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002640
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002640
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002616
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002616
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002616
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002628
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002628
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002628
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_109.zip
Uploading the intermediate file: upload_file_109.zip
  19:23:00 STEP 2641 H=2641:00 +CPU= 15.160
  19:23:15 STEP 2642 H=2642:00 +CPU= 15.589
  19:23:40 STEP 2643 H=2643:00 +CPU= 24.391
  19:23:55 STEP 2644 H=2644:00 +CPU= 15.755
  19:24:10 STEP 2645 H=2645:00 +CPU= 15.201
  19:24:35 STEP 2646 H=2646:00 +CPU= 24.806
  19:24:50 STEP 2647 H=2647:00 +CPU= 14.303
  19:25:01 STEP 2648 H=2648:00 +CPU= 11.388
  19:25:23 STEP 2649 H=2649:00 +CPU= 21.621
  19:25:38 STEP 2650 H=2650:00 +CPU= 15.829
  19:25:53 STEP 2651 H=2651:00 +CPU= 14.517
  19:26:17 STEP 2652 H=2652:00 +CPU= 23.643
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002652
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002652
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002652
  19:26:33 STEP 2653 H=2653:00 +CPU= 15.905
  19:26:49 STEP 2654 H=2654:00 +CPU= 16.375
  19:27:14 STEP 2655 H=2655:00 +CPU= 25.132
  19:27:30 STEP 2656 H=2656:00 +CPU= 16.245
  19:27:47 STEP 2657 H=2657:00 +CPU= 16.193
  19:28:12 STEP 2658 H=2658:00 +CPU= 25.587
  19:28:28 STEP 2659 H=2659:00 +CPU= 15.811
  19:28:44 STEP 2660 H=2660:00 +CPU= 16.353
  19:29:09 STEP 2661 H=2661:00 +CPU= 25.083
  19:29:25 STEP 2662 H=2662:00 +CPU= 16.009
  19:29:41 STEP 2663 H=2663:00 +CPU= 15.921
  19:30:10 STEP 2664 H=2664:00 +CPU= 28.336
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002664
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002664
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002664
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002640
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002640
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002640
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002652
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002652
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002652
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_110.zip
Uploading the intermediate file: upload_file_110.zip
  19:30:24 STEP 2665 H=2665:00 +CPU= 14.324
  19:30:36 STEP 2666 H=2666:00 +CPU= 11.669
  19:30:53 STEP 2667 H=2667:00 +CPU= 17.531
  19:31:04 STEP 2668 H=2668:00 +CPU= 11.189
  19:31:15 STEP 2669 H=2669:00 +CPU= 10.956
  19:31:33 STEP 2670 H=2670:00 +CPU= 17.876
  19:31:44 STEP 2671 H=2671:00 +CPU= 11.101
  19:31:55 STEP 2672 H=2672:00 +CPU= 11.090
  19:32:13 STEP 2673 H=2673:00 +CPU= 17.765
  19:32:24 STEP 2674 H=2674:00 +CPU= 11.234
  19:32:35 STEP 2675 H=2675:00 +CPU= 10.818
  19:32:55 STEP 2676 H=2676:00 +CPU= 19.487
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002676
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002676
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002676
  19:33:06 STEP 2677 H=2677:00 +CPU= 10.952
  19:33:17 STEP 2678 H=2678:00 +CPU= 11.148
  19:33:34 STEP 2679 H=2679:00 +CPU= 17.406
  19:33:45 STEP 2680 H=2680:00 +CPU= 11.050
  19:33:57 STEP 2681 H=2681:00 +CPU= 11.343
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 963
(argv5) wuid: 12180318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/alez/boinc/slots/3
Project directory is: /home/alez/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318
..mkdir for temp folder for results failed
Copying: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/alez/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/alez/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a20796ad13e01e72b502acfad939bf5 to: /home/alez/boinc/slots/3/oifs_43r3_ps_0674_1994050100_123_963_12180318.zip
Unzipping the namelist zip file: /home/alez/boinc/slots/3/oifs_43r3_ps_0674_1994050100_123_963_12180318.zip
ic_ancil_file: ic_ancil_12180318
ifsdata_file: ifsdata_12180318
climate_data_file: clim_data_12180318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c16e39d2b4e900bcf8d164e711e8f7a0 to: /home/alez/boinc/slots/3/ic_ancil_12180318.zip
Unzipping the IC ancils zip file: /home/alez/boinc/slots/3/ic_ancil_12180318.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eec56bb61123254f40b1abf8eb374e52 to: /home/alez/boinc/slots/3/ifsdata/ifsdata_12180318.zip
Unzipping the ifsdata_zip file: /home/alez/boinc/slots/3/ifsdata/ifsdata_12180318.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/alez/boinc/slots/3/159l_2/clim_data_12180318.zip
Unzipping the climate data zip file: /home/alez/boinc/slots/3/159l_2/clim_data_12180318.zip
Checking for progress XML file: /home/alez/boinc/slots/3/progress_file_12180318.xml
Opened progress file ok : /home/alez/boinc/slots/3/progress_file_12180318.xml
-- Model is restarting --
Adjusting last_iter, 2682, to previous model restart step.
Creating progress file: /home/alez/boinc/slots/3/progress_file_12180318.xml
last_cpu_time: 49573
upload_file_number: 111
last_iter: 2663
last_upload: 9590400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231
The child process has been launched with process id: 8925
Executing the command: /home/alez/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8925:8925] [20230124:093534:1674549334:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  09:36:04 STEP 2664 H=2664:00 +CPU= 26.874
  09:36:17 STEP 2665 H=2665:00 +CPU= 12.966
  09:36:32 STEP 2666 H=2666:00 +CPU= 15.297
  09:36:56 STEP 2667 H=2667:00 +CPU= 23.996
  09:37:12 STEP 2668 H=2668:00 +CPU= 15.694
  09:37:27 STEP 2669 H=2669:00 +CPU= 15.190
  09:37:52 STEP 2670 H=2670:00 +CPU= 24.453
  09:38:07 STEP 2671 H=2671:00 +CPU= 15.137
  09:38:22 STEP 2672 H=2672:00 +CPU= 15.500
  09:38:46 STEP 2673 H=2673:00 +CPU= 24.036
  09:39:02 STEP 2674 H=2674:00 +CPU= 15.465
  09:39:17 STEP 2675 H=2675:00 +CPU= 15.061
  09:39:43 STEP 2676 H=2676:00 +CPU= 26.747
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002676
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002676
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002676
  09:39:59 STEP 2677 H=2677:00 +CPU= 15.221
  09:40:14 STEP 2678 H=2678:00 +CPU= 15.702
  09:40:38 STEP 2679 H=2679:00 +CPU= 23.916
  09:40:54 STEP 2680 H=2680:00 +CPU= 15.372
  09:41:09 STEP 2681 H=2681:00 +CPU= 15.023
  09:41:33 STEP 2682 H=2682:00 +CPU= 24.487
  09:41:48 STEP 2683 H=2683:00 +CPU= 14.973
  09:42:04 STEP 2684 H=2684:00 +CPU= 15.761
  09:42:28 STEP 2685 H=2685:00 +CPU= 23.665
  09:42:43 STEP 2686 H=2686:00 +CPU= 15.520
  09:42:59 STEP 2687 H=2687:00 +CPU= 15.589
  09:43:27 STEP 2688 H=2688:00 +CPU= 28.059
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002688
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002688
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002688
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002664
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002664
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002664
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002676
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002676
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002676
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_111.zip
Uploading the intermediate file: upload_file_111.zip
  09:43:42 STEP 2689 H=2689:00 +CPU= 14.827
  09:43:57 STEP 2690 H=2690:00 +CPU= 15.386
  09:44:21 STEP 2691 H=2691:00 +CPU= 23.723
  09:44:37 STEP 2692 H=2692:00 +CPU= 15.727
  09:44:52 STEP 2693 H=2693:00 +CPU= 15.070
  09:45:16 STEP 2694 H=2694:00 +CPU= 24.459
  09:45:31 STEP 2695 H=2695:00 +CPU= 14.965
  09:45:47 STEP 2696 H=2696:00 +CPU= 15.950
  09:46:11 STEP 2697 H=2697:00 +CPU= 24.058
  09:46:27 STEP 2698 H=2698:00 +CPU= 15.678
  09:46:42 STEP 2699 H=2699:00 +CPU= 15.537
  09:47:05 STEP 2700 H=2700:00 +CPU= 23.208
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002700
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002700
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002700
  09:47:17 STEP 2701 H=2701:00 +CPU= 11.001
  09:47:28 STEP 2702 H=2702:00 +CPU= 11.429
  09:47:46 STEP 2703 H=2703:00 +CPU= 17.803
  09:47:57 STEP 2704 H=2704:00 +CPU= 11.320
  09:48:08 STEP 2705 H=2705:00 +CPU= 11.059
  09:48:31 STEP 2706 H=2706:00 +CPU= 22.552
  09:48:46 STEP 2707 H=2707:00 +CPU= 15.492
  09:49:02 STEP 2708 H=2708:00 +CPU= 16.170
  09:49:27 STEP 2709 H=2709:00 +CPU= 24.719
  09:49:43 STEP 2710 H=2710:00 +CPU= 16.005
  09:49:59 STEP 2711 H=2711:00 +CPU= 16.112
  09:50:28 STEP 2712 H=2712:00 +CPU= 28.970
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002712
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002712
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002712
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002688
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002688
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002688
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002700
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002700
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002700
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_112.zip
Uploading the intermediate file: upload_file_112.zip
  09:50:44 STEP 2713 H=2713:00 +CPU= 15.443
  09:50:59 STEP 2714 H=2714:00 +CPU= 15.850
  09:51:24 STEP 2715 H=2715:00 +CPU= 24.879
  09:51:40 STEP 2716 H=2716:00 +CPU= 15.625
  09:51:51 STEP 2717 H=2717:00 +CPU= 10.749
  09:52:08 STEP 2718 H=2718:00 +CPU= 17.737
  09:52:19 STEP 2719 H=2719:00 +CPU= 11.063
  09:52:31 STEP 2720 H=2720:00 +CPU= 11.137
  09:52:48 STEP 2721 H=2721:00 +CPU= 17.530
  09:52:59 STEP 2722 H=2722:00 +CPU= 11.291
  09:53:10 STEP 2723 H=2723:00 +CPU= 10.673
  09:53:30 STEP 2724 H=2724:00 +CPU= 19.540
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002724
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002724
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002724
  09:53:40 STEP 2725 H=2725:00 +CPU= 10.803
  09:53:52 STEP 2726 H=2726:00 +CPU= 11.251
  09:54:09 STEP 2727 H=2727:00 +CPU= 17.230
  09:54:20 STEP 2728 H=2728:00 +CPU= 11.123
  09:54:31 STEP 2729 H=2729:00 +CPU= 10.719
  09:54:49 STEP 2730 H=2730:00 +CPU= 18.312
  09:55:04 STEP 2731 H=2731:00 +CPU= 14.906
  09:55:20 STEP 2732 H=2732:00 +CPU= 15.637
  09:55:43 STEP 2733 H=2733:00 +CPU= 23.063
  09:55:59 STEP 2734 H=2734:00 +CPU= 16.282
  09:56:15 STEP 2735 H=2735:00 +CPU= 16.213
  09:56:45 STEP 2736 H=2736:00 +CPU= 29.179
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002736
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002736
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002736
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002712
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002712
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002712
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002724
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002724
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002724
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_113.zip
Uploading the intermediate file: upload_file_113.zip
  09:57:00 STEP 2737 H=2737:00 +CPU= 15.611
  09:57:16 STEP 2738 H=2738:00 +CPU= 16.040
  09:57:41 STEP 2739 H=2739:00 +CPU= 24.759
  09:57:57 STEP 2740 H=2740:00 +CPU= 16.121
  09:58:13 STEP 2741 H=2741:00 +CPU= 15.484
  09:58:38 STEP 2742 H=2742:00 +CPU= 25.325
  09:58:52 STEP 2743 H=2743:00 +CPU= 13.531
  09:59:08 STEP 2744 H=2744:00 +CPU= 16.201
  09:59:33 STEP 2745 H=2745:00 +CPU= 24.745
  09:59:49 STEP 2746 H=2746:00 +CPU= 16.291
  10:00:05 STEP 2747 H=2747:00 +CPU= 15.807
  10:00:33 STEP 2748 H=2748:00 +CPU= 27.976
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002748
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002748
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002748
  10:00:48 STEP 2749 H=2749:00 +CPU= 15.487
  10:01:05 STEP 2750 H=2750:00 +CPU= 16.329
  10:01:29 STEP 2751 H=2751:00 +CPU= 24.510
  10:01:45 STEP 2752 H=2752:00 +CPU= 16.199
  10:02:01 STEP 2753 H=2753:00 +CPU= 15.463
  10:02:26 STEP 2754 H=2754:00 +CPU= 25.366
  10:02:42 STEP 2755 H=2755:00 +CPU= 15.939
  10:02:58 STEP 2756 H=2756:00 +CPU= 15.976
  10:03:23 STEP 2757 H=2757:00 +CPU= 24.766
  10:03:39 STEP 2758 H=2758:00 +CPU= 16.383
  10:03:55 STEP 2759 H=2759:00 +CPU= 16.140
  10:04:23 STEP 2760 H=2760:00 +CPU= 27.231
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002760
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002760
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002760
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002736
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002736
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002736
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002748
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002748
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002748
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_114.zip
Uploading the intermediate file: upload_file_114.zip
  10:04:33 STEP 2761 H=2761:00 +CPU= 10.827
  10:04:46 STEP 2762 H=2762:00 +CPU= 12.477
  10:05:07 STEP 2763 H=2763:00 +CPU= 20.818
  10:05:23 STEP 2764 H=2764:00 +CPU= 15.642
  10:05:38 STEP 2765 H=2765:00 +CPU= 14.976
  10:06:03 STEP 2766 H=2766:00 +CPU= 24.439
  10:06:18 STEP 2767 H=2767:00 +CPU= 15.161
  10:06:33 STEP 2768 H=2768:00 +CPU= 15.528
  10:06:57 STEP 2769 H=2769:00 +CPU= 24.009
  10:07:13 STEP 2770 H=2770:00 +CPU= 15.726
  10:07:28 STEP 2771 H=2771:00 +CPU= 15.278
  10:07:55 STEP 2772 H=2772:00 +CPU= 26.898
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002772
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002772
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002772
  10:08:10 STEP 2773 H=2773:00 +CPU= 15.044
  10:08:26 STEP 2774 H=2774:00 +CPU= 15.591
  10:08:50 STEP 2775 H=2775:00 +CPU= 24.083
  10:09:06 STEP 2776 H=2776:00 +CPU= 15.934
  10:09:21 STEP 2777 H=2777:00 +CPU= 15.189
  10:09:46 STEP 2778 H=2778:00 +CPU= 24.496
  10:10:01 STEP 2779 H=2779:00 +CPU= 15.076
  10:10:16 STEP 2780 H=2780:00 +CPU= 15.617
  10:10:41 STEP 2781 H=2781:00 +CPU= 24.266
  10:10:56 STEP 2782 H=2782:00 +CPU= 15.565
  10:11:12 STEP 2783 H=2783:00 +CPU= 15.608
  10:11:40 STEP 2784 H=2784:00 +CPU= 28.262
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002784
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002784
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002784
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002760
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002760
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002760
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002772
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002772
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002772
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_115.zip
Uploading the intermediate file: upload_file_115.zip
  10:11:55 STEP 2785 H=2785:00 +CPU= 15.113
  10:12:11 STEP 2786 H=2786:00 +CPU= 15.849
  10:12:35 STEP 2787 H=2787:00 +CPU= 24.274
  10:12:51 STEP 2788 H=2788:00 +CPU= 15.577
  10:13:06 STEP 2789 H=2789:00 +CPU= 15.054
  10:13:31 STEP 2790 H=2790:00 +CPU= 24.887
  10:13:46 STEP 2791 H=2791:00 +CPU= 15.144
  10:14:02 STEP 2792 H=2792:00 +CPU= 15.678
  10:14:26 STEP 2793 H=2793:00 +CPU= 24.375
  10:14:42 STEP 2794 H=2794:00 +CPU= 15.833
  10:14:57 STEP 2795 H=2795:00 +CPU= 15.384
  10:15:24 STEP 2796 H=2796:00 +CPU= 27.117
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002796
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002796
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002796
  10:15:40 STEP 2797 H=2797:00 +CPU= 15.234
  10:15:55 STEP 2798 H=2798:00 +CPU= 15.757
  10:16:20 STEP 2799 H=2799:00 +CPU= 24.237
  10:16:35 STEP 2800 H=2800:00 +CPU= 15.740
  10:16:48 STEP 2801 H=2801:00 +CPU= 12.390
  10:17:12 STEP 2802 H=2802:00 +CPU= 23.898
  10:17:23 STEP 2803 H=2803:00 +CPU= 10.928
  10:17:34 STEP 2804 H=2804:00 +CPU= 11.652
  10:17:52 STEP 2805 H=2805:00 +CPU= 17.720
  10:18:04 STEP 2806 H=2806:00 +CPU= 12.133
  10:18:20 STEP 2807 H=2807:00 +CPU= 16.363
  10:18:50 STEP 2808 H=2808:00 +CPU= 29.249
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002808
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002808
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002808
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002784
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002784
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002784
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002796
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002796
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002796
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_116.zip
Uploading the intermediate file: upload_file_116.zip
  10:19:05 STEP 2809 H=2809:00 +CPU= 15.502
  10:19:21 STEP 2810 H=2810:00 +CPU= 16.304
  10:19:46 STEP 2811 H=2811:00 +CPU= 24.754
  10:20:03 STEP 2812 H=2812:00 +CPU= 16.435
  10:20:18 STEP 2813 H=2813:00 +CPU= 15.549
  10:20:44 STEP 2814 H=2814:00 +CPU= 25.534
  10:20:59 STEP 2815 H=2815:00 +CPU= 15.672
  10:21:16 STEP 2816 H=2816:00 +CPU= 16.258
  10:21:40 STEP 2817 H=2817:00 +CPU= 24.785
  10:21:57 STEP 2818 H=2818:00 +CPU= 16.317
  10:22:13 STEP 2819 H=2819:00 +CPU= 15.995
  10:22:41 STEP 2820 H=2820:00 +CPU= 27.825
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002820
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002820
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002820
  10:22:56 STEP 2821 H=2821:00 +CPU= 15.691
  10:23:13 STEP 2822 H=2822:00 +CPU= 16.342
  10:23:38 STEP 2823 H=2823:00 +CPU= 24.948
  10:23:54 STEP 2824 H=2824:00 +CPU= 16.017
  10:24:09 STEP 2825 H=2825:00 +CPU= 15.493
  10:24:35 STEP 2826 H=2826:00 +CPU= 25.456
  10:24:50 STEP 2827 H=2827:00 +CPU= 15.770
  10:25:07 STEP 2828 H=2828:00 +CPU= 16.385
  10:25:32 STEP 2829 H=2829:00 +CPU= 24.939
  10:25:48 STEP 2830 H=2830:00 +CPU= 16.101
  10:26:04 STEP 2831 H=2831:00 +CPU= 16.178
  10:26:33 STEP 2832 H=2832:00 +CPU= 28.801
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002832
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002832
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002832
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002808
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002808
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002808
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002820
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002820
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002820
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_117.zip
Uploading the intermediate file: upload_file_117.zip
  10:26:48 STEP 2833 H=2833:00 +CPU= 15.646
  10:27:04 STEP 2834 H=2834:00 +CPU= 15.865
  10:27:29 STEP 2835 H=2835:00 +CPU= 24.596
  10:27:45 STEP 2836 H=2836:00 +CPU= 15.916
  10:28:00 STEP 2837 H=2837:00 +CPU= 15.378
  10:28:25 STEP 2838 H=2838:00 +CPU= 25.218
  10:28:41 STEP 2839 H=2839:00 +CPU= 15.850
  10:28:57 STEP 2840 H=2840:00 +CPU= 16.231
  10:29:22 STEP 2841 H=2841:00 +CPU= 24.473
  10:29:38 STEP 2842 H=2842:00 +CPU= 16.108
  10:29:54 STEP 2843 H=2843:00 +CPU= 15.631
  10:30:15 STEP 2844 H=2844:00 +CPU= 21.604
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002844
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002844
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002844
  10:30:27 STEP 2845 H=2845:00 +CPU= 11.357
  10:30:38 STEP 2846 H=2846:00 +CPU= 11.639
  10:30:56 STEP 2847 H=2847:00 +CPU= 18.111
  10:31:08 STEP 2848 H=2848:00 +CPU= 11.763
  10:31:19 STEP 2849 H=2849:00 +CPU= 11.260
  10:31:38 STEP 2850 H=2850:00 +CPU= 18.101
  10:31:49 STEP 2851 H=2851:00 +CPU= 11.236
  10:32:00 STEP 2852 H=2852:00 +CPU= 11.347
  10:32:18 STEP 2853 H=2853:00 +CPU= 17.923
  10:32:30 STEP 2854 H=2854:00 +CPU= 11.673
  10:32:41 STEP 2855 H=2855:00 +CPU= 11.710
  10:33:10 STEP 2856 H=2856:00 +CPU= 28.285
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002856
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002856
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002856
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002832
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002832
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002832
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMGGhq0f+002844
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMSHhq0f+002844
Adding to the zip: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318/ICMUAhq0f+002844
Zipping up the intermediate file: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231_118.zip
Uploading the intermediate file: upload_file_118.zip
  10:33:25 STEP 2857 H=2857:00 +CPU= 15.266
  10:33:41 STEP 2858 H=2858:00 +CPU= 16.019
  10:34:06 STEP 2859 H=2859:00 +CPU= 24.400
  10:34:22 STEP 2860 H=2860:00 +CPU= 15.892
  10:34:37 STEP 2861 H=2861:00 +CPU= 15.392
  10:35:02 STEP 2862 H=2862:00 +CPU= 25.073
  10:35:18 STEP 2863 H=2863:00 +CPU= 15.728
  10:35:34 STEP 2864 H=2864:00 +CPU= 15.975
  10:35:58 STEP 2865 H=2865:00 +CPU= 24.275
  10:36:14 STEP 2866 H=2866:00 +CPU= 15.953
  10:36:29 STEP 2867 H=2867:00 +CPU= 15.386
Suspend request received from the BOINC client, suspending the child process
Quit request received from the BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1994050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0674
(argv4) batchid: 963
(argv5) wuid: 12180318
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /home/alez/boinc/slots/3
Project directory is: /home/alez/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_12180318
..mkdir for temp folder for results failed
Copying: /home/alez/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /home/alez/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /home/alez/boinc/slots/3/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0a20796ad13e01e72b502acfad939bf5 to: /home/alez/boinc/slots/3/oifs_43r3_ps_0674_1994050100_123_963_12180318.zip
Unzipping the namelist zip file: /home/alez/boinc/slots/3/oifs_43r3_ps_0674_1994050100_123_963_12180318.zip
ic_ancil_file: ic_ancil_12180318
ifsdata_file: ifsdata_12180318
climate_data_file: clim_data_12180318
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_c16e39d2b4e900bcf8d164e711e8f7a0 to: /home/alez/boinc/slots/3/ic_ancil_12180318.zip
Unzipping the IC ancils zip file: /home/alez/boinc/slots/3/ic_ancil_12180318.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_eec56bb61123254f40b1abf8eb374e52 to: /home/alez/boinc/slots/3/ifsdata/ifsdata_12180318.zip
Unzipping the ifsdata_zip file: /home/alez/boinc/slots/3/ifsdata/ifsdata_12180318.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /home/alez/boinc/slots/3/159l_2/clim_data_12180318.zip
Unzipping the climate data zip file: /home/alez/boinc/slots/3/159l_2/clim_data_12180318.zip
Checking for progress XML file: /home/alez/boinc/slots/3/progress_file_12180318.xml
Opened progress file ok : /home/alez/boinc/slots/3/progress_file_12180318.xml
-- Model is restarting --
Adjusting last_iter, 2867, to previous model restart step.
Creating progress file: /home/alez/boinc/slots/3/progress_file_12180318.xml
last_cpu_time: 53217
upload_file_number: 119
last_iter: 2855
last_upload: 10281600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0674_1994050100_123_963_12180318_0_r551073231
The child process has been launched with process id: 8687
Executing the command: /home/alez/boinc/slots/3/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:ichigo:1:1:8687:8687] [20230125:212201:1674678121:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:22:31 STEP 2856 H=2856:00 +CPU= 27.359
  21:22:42 STEP 2857 H=2857:00 +CPU= 10.989
  21:22:53 STEP 2858 H=2858:00 +CPU= 11.016
  21:23:11 STEP 2859 H=2859:00 +CPU= 17.782
  21:23:22 STEP 2860 H=2860:00 +CPU= 11.012
  21:23:33 STEP 2861 H=2861:00 +CPU= 10.868
  21:23:51 STEP 2862 H=2862:00 +CPU= 17.891
  21:24:01 STEP 2863 H=2863:00 +CPU= 10.734
  21:24:13 STEP 2864 H=2864:00 +CPU= 11.318
  21:24:32 STEP 2865 H=2865:00 +CPU= 19.476
  21:24:48 STEP 2866 H=2866:00 +CPU= 15.650
  21:25:03 STEP 2867 H=2867:00 +CPU= 15.163
  21:25:30 STEP 2868 H=2868:00 +CPU= 26.860
Moving to projects directory: /home/alez/boinc/slots/3/ICMGGhq0f+002868
Moving to projects directory: /home/alez/boinc/slots/3/ICMSHhq0f+002868
Moving to projects directory: /home/alez/boinc/slots/3/ICMUAhq0f+002868
  21:25:45 STEP 2869 H=2869:00 +CPU= 15.087
  21:26:01 STEP 2870 H=2870:00 +CPU= 15.715
  21:26:25 STEP 2871 H=2871:00 +CPU= 24.287
  21:26:41 STEP 2872 H=2872:00 +CPU= 15.748
  21:26:56 STEP 2873 H=2873:00 +CPU= 15.028
  21:27:21 STEP 2874 H=2874:00 +CPU= 24.978
  21:27:36 STEP 2875 H=2875:00 +CPU= 15.183
  21:27:52 STEP 2876 H=2876:00 +CPU= 15.784
  21:28:16 STEP 2877 H=2877:00 +CPU= 24.264
  21:28:32 STEP 2878 H=2878:00 +CPU= 15.926
  21:28:48 STEP 2879 H=2879:00 +CPU= 15.623
  21:29:16 STEP 2880 H=2880:00 +CPU= 28.183
Abort request received from BOINC client, ending the child process
..Failed, model did not complete successfully

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:38:06 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 9,417,600 48,429 0.0051
09 May 2023 16:38:00 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 8,553,600 44,158 0.0052
09 May 2023 16:37:55 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 7,689,600 39,867 0.0052
09 May 2023 16:37:44 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 6,825,600 35,395 0.0052
09 May 2023 16:37:40 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 5,961,600 31,156 0.0052
09 May 2023 16:36:02 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 4,924,800 25,726 0.0052
09 May 2023 16:35:53 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 4,060,800 21,686 0.0053
09 May 2023 16:35:38 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 2,419,200 12,837 0.0053
09 May 2023 16:35:32 1410588 22265277 oifs_43r3_ps_0674_1994050100_123_963_12180318_0 1,555,200 8,519 0.0055


©2024 cpdn.org