climateprediction.net home page
Task 22261421

Task 22261421

Name oifs_43r3_ps_0018_1991050100_123_960_12176662_0
Workunit 12176662
Created 21 Dec 2022, 20:42:16 UTC
Sent 21 Dec 2022, 20:48:34 UTC
Report deadline 20 Jan 2023, 20:48:34 UTC
Received 26 Dec 2022, 16:18:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 148 (0x00000094) Unknown error code
Computer ID 1534746
Run time 14 hours 30 min 7 sec
CPU time 14 hours 30 min 7 sec
Validate state Invalid
Credit 12,096.00
Device peak FLOPS 5.61 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,795.16 MB
Peak swap size 4,974.32 MB
Peak disk usage 1,265.37 MB
Stderr 
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 148 (0x94, -108)</message>
<stderr_txt>
_ps_12176662/ICMGGhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002424
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002436
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002436
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422_101.zip
Uploading the intermediate file: upload_file_101.zip
  17:21:29 STEP 2449 H=2449:00 +CPU= 17.959
  17:21:46 STEP 2450 H=2450:00 +CPU= 17.456
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:22:18 STEP 2451 H=2451:00 +CPU= 28.850
  17:22:37 STEP 2452 H=2452:00 +CPU= 18.705
  17:22:55 STEP 2453 H=2453:00 +CPU= 17.205
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:23:26 STEP 2454 H=2454:00 +CPU= 29.118
  17:23:43 STEP 2455 H=2455:00 +CPU= 16.782
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:04 STEP 2456 H=2456:00 +CPU= 19.227
  17:24:33 STEP 2457 H=2457:00 +CPU= 28.482
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:24:53 STEP 2458 H=2458:00 +CPU= 18.151
  17:25:13 STEP 2459 H=2459:00 +CPU= 18.438
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:25:47 STEP 2460 H=2460:00 +CPU= 32.421
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002460
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002460
  17:26:04 STEP 2461 H=2461:00 +CPU= 17.032
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:26:24 STEP 2462 H=2462:00 +CPU= 17.013
  17:26:53 STEP 2463 H=2463:00 +CPU= 28.758
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:27:14 STEP 2464 H=2464:00 +CPU= 19.321
  17:27:33 STEP 2465 H=2465:00 +CPU= 18.675
  17:28:04 STEP 2466 H=2466:00 +CPU= 29.756
  17:28:22 STEP 2467 H=2467:00 +CPU= 17.581
  17:28:41 STEP 2468 H=2468:00 +CPU= 18.799
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:29:14 STEP 2469 H=2469:00 +CPU= 29.203
  17:29:33 STEP 2470 H=2470:00 +CPU= 19.043
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:29:57 STEP 2471 H=2471:00 +CPU= 21.041
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:30:33 STEP 2472 H=2472:00 +CPU= 33.857
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002472
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002448
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002460
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002460
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422_102.zip
Uploading the intermediate file: upload_file_102.zip
  17:30:51 STEP 2473 H=2473:00 +CPU= 17.157
  17:31:10 STEP 2474 H=2474:00 +CPU= 18.867
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:31:42 STEP 2475 H=2475:00 +CPU= 30.203
  17:32:01 STEP 2476 H=2476:00 +CPU= 18.663
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:32:22 STEP 2477 H=2477:00 +CPU= 18.794
  17:32:53 STEP 2478 H=2478:00 +CPU= 30.466
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:33:15 STEP 2479 H=2479:00 +CPU= 20.413
  17:33:35 STEP 2480 H=2480:00 +CPU= 19.840
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:34:07 STEP 2481 H=2481:00 +CPU= 29.608
  17:34:26 STEP 2482 H=2482:00 +CPU= 18.606
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:34:48 STEP 2483 H=2483:00 +CPU= 19.471
  17:35:21 STEP 2484 H=2484:00 +CPU= 33.275
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002484
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002484
  17:35:42 STEP 2485 H=2485:00 +CPU= 19.286
  17:36:02 STEP 2486 H=2486:00 +CPU= 20.145
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:36:36 STEP 2487 H=2487:00 +CPU= 31.178
  17:36:55 STEP 2488 H=2488:00 +CPU= 18.648
  17:37:14 STEP 2489 H=2489:00 +CPU= 19.221
  17:37:49 STEP 2490 H=2490:00 +CPU= 30.488
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:38:11 STEP 2491 H=2491:00 +CPU= 20.282
  17:38:32 STEP 2492 H=2492:00 +CPU= 20.701
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:39:06 STEP 2493 H=2493:00 +CPU= 31.596
  17:39:26 STEP 2494 H=2494:00 +CPU= 19.859
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:39:49 STEP 2495 H=2495:00 +CPU= 20.829
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:40:28 STEP 2496 H=2496:00 +CPU= 35.840
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002496
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002472
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002484
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002484
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422_103.zip
Uploading the intermediate file: upload_file_103.zip
  17:40:47 STEP 2497 H=2497:00 +CPU= 18.551
  17:41:06 STEP 2498 H=2498:00 +CPU= 18.921
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:41:40 STEP 2499 H=2499:00 +CPU= 32.116
  17:42:00 STEP 2500 H=2500:00 +CPU= 19.848
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:42:22 STEP 2501 H=2501:00 +CPU= 19.188
  17:42:52 STEP 2502 H=2502:00 +CPU= 29.717
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:43:14 STEP 2503 H=2503:00 +CPU= 20.446
  17:43:35 STEP 2504 H=2504:00 +CPU= 20.005
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:44:06 STEP 2505 H=2505:00 +CPU= 29.796
  17:44:26 STEP 2506 H=2506:00 +CPU= 18.958
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:44:47 STEP 2507 H=2507:00 +CPU= 19.501
  17:45:20 STEP 2508 H=2508:00 +CPU= 31.901
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002508
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002508
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:45:41 STEP 2509 H=2509:00 +CPU= 19.387
  17:46:00 STEP 2510 H=2510:00 +CPU= 18.938
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:46:33 STEP 2511 H=2511:00 +CPU= 31.259
  17:46:53 STEP 2512 H=2512:00 +CPU= 19.772
  17:47:12 STEP 2513 H=2513:00 +CPU= 18.655
  17:47:42 STEP 2514 H=2514:00 +CPU= 30.347
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:48:03 STEP 2515 H=2515:00 +CPU= 18.898
  17:48:22 STEP 2516 H=2516:00 +CPU= 18.947
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:48:56 STEP 2517 H=2517:00 +CPU= 31.439
  17:49:14 STEP 2518 H=2518:00 +CPU= 18.537
  17:49:33 STEP 2519 H=2519:00 +CPU= 18.535
  17:50:09 STEP 2520 H=2520:00 +CPU= 35.955
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002520
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002496
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002508
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002508
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422_104.zip
Uploading the intermediate file: upload_file_104.zip
  17:50:29 STEP 2521 H=2521:00 +CPU= 19.740
  17:50:48 STEP 2522 H=2522:00 +CPU= 19.792
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:51:22 STEP 2523 H=2523:00 +CPU= 31.583
  17:51:43 STEP 2524 H=2524:00 +CPU= 20.871
  17:52:02 STEP 2525 H=2525:00 +CPU= 19.030
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:52:34 STEP 2526 H=2526:00 +CPU= 30.360
  17:52:53 STEP 2527 H=2527:00 +CPU= 18.391
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:53:15 STEP 2528 H=2528:00 +CPU= 19.832
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:53:48 STEP 2529 H=2529:00 +CPU= 31.390
  17:54:08 STEP 2530 H=2530:00 +CPU= 20.290
  17:54:27 STEP 2531 H=2531:00 +CPU= 18.364
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:55:02 STEP 2532 H=2532:00 +CPU= 32.390
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002532
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002532
  17:55:21 STEP 2533 H=2533:00 +CPU= 18.939
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:55:42 STEP 2534 H=2534:00 +CPU= 18.668
  17:56:10 STEP 2535 H=2535:00 +CPU= 28.375
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:56:34 STEP 2536 H=2536:00 +CPU= 20.440
  17:56:53 STEP 2537 H=2537:00 +CPU= 19.716
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:57:27 STEP 2538 H=2538:00 +CPU= 31.347
  17:57:46 STEP 2539 H=2539:00 +CPU= 18.769
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:58:08 STEP 2540 H=2540:00 +CPU= 19.825
  17:58:38 STEP 2541 H=2541:00 +CPU= 30.702
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:59:00 STEP 2542 H=2542:00 +CPU= 20.155
  17:59:19 STEP 2543 H=2543:00 +CPU= 18.518
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  17:59:57 STEP 2544 H=2544:00 +CPU= 35.025
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002544
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002544
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002520
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMGGhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMSHhq0f+002532
Adding to the zip: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662/ICMUAhq0f+002532
Zipping up the intermediate file: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422_105.zip
Uploading the intermediate file: upload_file_105.zip
  18:00:15 STEP 2545 H=2545:00 +CPU= 18.763
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:00:37 STEP 2546 H=2546:00 +CPU= 19.729
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:01:21 STEP 2547 H=2547:00 +CPU= 30.311
  18:01:42 STEP 2548 H=2548:00 +CPU= 20.904
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:07:19 STEP 2549 H=2549:00 +CPU= 22.816
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:07:59 STEP 2550 H=2550:00 +CPU= 33.353
  18:08:19 STEP 2551 H=2551:00 +CPU= 19.606
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:08:41 STEP 2552 H=2552:00 +CPU= 19.492
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:09:13 STEP 2553 H=2553:00 +CPU= 30.183
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:13:03 STEP 2554 H=2554:00 +CPU= 23.702
  18:13:22 STEP 2555 H=2555:00 +CPU= 18.417
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:13:58 STEP 2556 H=2556:00 +CPU= 33.863
Moving to projects directory: /var/lib/boinc/slots/19/ICMGGhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/19/ICMSHhq0f+002556
Moving to projects directory: /var/lib/boinc/slots/19/ICMUAhq0f+002556
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:14:21 STEP 2557 H=2557:00 +CPU= 19.825
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
  18:46:28 STEP 2558 H=2558:00 +CPU= 29.603
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8d4d53f4eb835dbb5d4ebb1da099633e to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Checking for progress XML file: /var/lib/boinc/slots/19/progress_file_12176662.xml
Opened progress file ok : /var/lib/boinc/slots/19/progress_file_12176662.xml
-- Model is restarting --
Adjusting last_iter, 2557, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/19/progress_file_12176662.xml
last_cpu_time: 48947
upload_file_number: 106
last_iter: 2543
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422
The child process has been launched with process id: 14902
Executing the command: /var/lib/boinc/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.034] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:14902:14902] [20221222:184656:1671709616:0.043] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
  18:47:08 STEP 2559 H=2559:00 +CPU= 33.784
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8d4d53f4eb835dbb5d4ebb1da099633e to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Checking for progress XML file: /var/lib/boinc/slots/19/progress_file_12176662.xml
Opened progress file ok : /var/lib/boinc/slots/19/progress_file_12176662.xml
-- Model is restarting --
Adjusting last_iter, 2560, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/19/progress_file_12176662.xml
last_cpu_time: 50389
upload_file_number: 106
last_iter: 2543
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422
The child process has been launched with process id: 20027
Executing the command: /var/lib/boinc/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.103] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:20027:20027] [20221225:142436:1671953076:0.145] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
  05:15:00 STEP 2544 H=2544:00 +CPU=547.790
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8d4d53f4eb835dbb5d4ebb1da099633e to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Checking for progress XML file: /var/lib/boinc/slots/19/progress_file_12176662.xml
Opened progress file ok : /var/lib/boinc/slots/19/progress_file_12176662.xml
-- Model is restarting --
Adjusting last_iter, -999, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/19/progress_file_12176662.xml
last_cpu_time: 51838
upload_file_number: 106
last_iter: -985
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422
The child process has been launched with process id: 21892
Executing the command: /var/lib/boinc/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.008] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.012] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:21892:21892] [20221226:051529:1672006529:0.016] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:15:29 STEP 2545 H=2545:00 +CPU= 24.915
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
Suspend request received from the BOINC client, suspending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
  07:33:13 STEP 2546 H=2546:00 +CPU=797.220
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
  07:36:05 STEP 2547 H=2547:00 +CPU= 49.191
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8d4d53f4eb835dbb5d4ebb1da099633e to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
  20:12:28 STEP 2548 H=2548:00 +CPU=600.435
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8d4d53f4eb835dbb5d4ebb1da099633e to: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
Unzipping the ifsdata_zip file: /var/lib/boinc/slots/19/ifsdata/ifsdata_12176662.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Unzipping the climate data zip file: /var/lib/boinc/slots/19/159l_2/clim_data_12176662.zip
Checking for progress XML file: /var/lib/boinc/slots/19/progress_file_12176662.xml
Opened progress file ok : /var/lib/boinc/slots/19/progress_file_12176662.xml
-- Model is restarting --
Adjusting last_iter, 2544, to previous model restart step.
Creating progress file: /var/lib/boinc/slots/19/progress_file_12176662.xml
last_cpu_time: 52136
upload_file_number: 106
last_iter: 2543
last_upload: 9158400
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0018_1991050100_123_960_12176662_0_r1818012422
The child process has been launched with process id: 23079
Executing the command: /var/lib/boinc/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.038] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:vm9:1:1:23079:23079] [20221226:201257:1672060377:0.039] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  20:12:58 STEP 2549 H=2549:00 +CPU= 25.263
Suspend request received from the BOINC client, suspending the child process
Resuming the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_d087200953d9223950268b726b59e7f1 to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
Unzipping the IC ancils zip file: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
  22:03:52 STEP 2550 H=2550:00 +CPU=292.260
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1991050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0018
(argv4) batchid: 960
(argv5) wuid: 12176662
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc/slots/19
Project directory is: /var/lib/boinc/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_12176662
..mkdir for temp folder for results failed
Copying: /var/lib/boinc/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_13c6ad5f83f02eafb87924fb75d7b434 to: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
Unzipping the namelist zip file: /var/lib/boinc/slots/19/oifs_43r3_ps_0018_1991050100_123_960_12176662.zip
ic_ancil_file: ic_ancil_12176662
ifsdata_file: ifsdata_12176662
climate_data_file: clim_data_12176662
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from:  to: /var/lib/boinc/slots/19/ic_ancil_12176662.zip
..Copying the IC ancils to the working directory failed
  22:15:10 STEP 2551 H=2551:00 +CPU=166.449

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 15:47:49 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 8,640,000 45,406 0.0053
09 May 2023 15:47:39 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 7,776,000 40,357 0.0052
09 May 2023 15:47:33 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 6,912,000 35,619 0.0052
09 May 2023 15:47:22 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 6,048,000 31,007 0.0051
09 May 2023 15:47:16 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 5,184,000 26,451 0.0051
09 May 2023 15:47:11 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 4,320,000 21,878 0.0051
09 May 2023 15:47:04 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 3,456,000 17,275 0.0050
09 May 2023 15:46:54 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 2,592,000 12,878 0.0050
09 May 2023 15:46:48 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 1,728,000 8,579 0.0050
09 May 2023 15:46:43 1534746 22261421 oifs_43r3_ps_0018_1991050100_123_960_12176662_0 864,000 4,298 0.0050


©2024 climateprediction.net