<core_client_version>7.16.6</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)</message>
<stderr_txt>
ediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002364
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002364
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_42.zip
Uploading the intermediate file: upload_file_42.zip
Uploading trickle at timestep: 8553600
15:53:01 STEP 2377 H=2377:00 +CPU= 26.006
15:53:27 STEP 2378 H=2378:00 +CPU= 25.979
15:54:00 STEP 2379 H=2379:00 +CPU= 33.007
15:54:24 STEP 2380 H=2380:00 +CPU= 23.585
15:54:52 STEP 2381 H=2381:00 +CPU= 27.763
15:55:30 STEP 2382 H=2382:00 +CPU= 38.482
15:55:51 STEP 2383 H=2383:00 +CPU= 20.972
15:56:11 STEP 2384 H=2384:00 +CPU= 19.885
15:56:42 STEP 2385 H=2385:00 +CPU= 30.465
15:57:05 STEP 2386 H=2386:00 +CPU= 23.908
15:57:29 STEP 2387 H=2387:00 +CPU= 23.384
15:58:07 STEP 2388 H=2388:00 +CPU= 38.596
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002388
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002388
15:58:32 STEP 2389 H=2389:00 +CPU= 25.026
15:58:55 STEP 2390 H=2390:00 +CPU= 22.901
15:59:27 STEP 2391 H=2391:00 +CPU= 31.787
15:59:50 STEP 2392 H=2392:00 +CPU= 22.508
16:00:10 STEP 2393 H=2393:00 +CPU= 20.292
16:00:47 STEP 2394 H=2394:00 +CPU= 36.600
16:01:16 STEP 2395 H=2395:00 +CPU= 28.878
16:01:45 STEP 2396 H=2396:00 +CPU= 29.879
16:02:17 STEP 2397 H=2397:00 +CPU= 31.326
16:02:38 STEP 2398 H=2398:00 +CPU= 21.731
16:03:06 STEP 2399 H=2399:00 +CPU= 27.619
16:03:53 STEP 2400 H=2400:00 +CPU= 46.208
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002400
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002376
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002388
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002388
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_43.zip
Uploading the intermediate file: upload_file_43.zip
16:04:13 STEP 2401 H=2401:00 +CPU= 19.821
Uploading trickle at timestep: 8640000
16:04:36 STEP 2402 H=2402:00 +CPU= 23.155
16:05:12 STEP 2403 H=2403:00 +CPU= 35.858
16:05:34 STEP 2404 H=2404:00 +CPU= 21.696
16:06:01 STEP 2405 H=2405:00 +CPU= 26.971
16:06:42 STEP 2406 H=2406:00 +CPU= 41.432
16:07:04 STEP 2407 H=2407:00 +CPU= 21.594
16:07:28 STEP 2408 H=2408:00 +CPU= 23.641
16:08:07 STEP 2409 H=2409:00 +CPU= 39.542
16:08:31 STEP 2410 H=2410:00 +CPU= 24.235
16:08:52 STEP 2411 H=2411:00 +CPU= 20.950
16:09:32 STEP 2412 H=2412:00 +CPU= 39.686
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002412
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002412
16:09:56 STEP 2413 H=2413:00 +CPU= 23.712
16:10:19 STEP 2414 H=2414:00 +CPU= 22.763
16:10:57 STEP 2415 H=2415:00 +CPU= 38.114
16:11:20 STEP 2416 H=2416:00 +CPU= 23.181
16:11:41 STEP 2417 H=2417:00 +CPU= 20.703
16:12:16 STEP 2418 H=2418:00 +CPU= 35.195
16:12:37 STEP 2419 H=2419:00 +CPU= 21.367
16:13:04 STEP 2420 H=2420:00 +CPU= 26.542
16:13:50 STEP 2421 H=2421:00 +CPU= 45.816
16:14:16 STEP 2422 H=2422:00 +CPU= 26.348
16:14:37 STEP 2423 H=2423:00 +CPU= 20.748
16:15:19 STEP 2424 H=2424:00 +CPU= 37.960
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002424
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002400
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002412
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002412
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_44.zip
Uploading the intermediate file: upload_file_44.zip
16:15:39 STEP 2425 H=2425:00 +CPU= 20.075
Uploading trickle at timestep: 8726400
16:16:00 STEP 2426 H=2426:00 +CPU= 20.559
16:16:31 STEP 2427 H=2427:00 +CPU= 30.584
16:16:53 STEP 2428 H=2428:00 +CPU= 22.130
16:17:13 STEP 2429 H=2429:00 +CPU= 20.552
16:17:49 STEP 2430 H=2430:00 +CPU= 35.438
16:18:11 STEP 2431 H=2431:00 +CPU= 22.170
16:18:34 STEP 2432 H=2432:00 +CPU= 22.676
16:19:07 STEP 2433 H=2433:00 +CPU= 33.240
16:19:30 STEP 2434 H=2434:00 +CPU= 22.732
16:19:52 STEP 2435 H=2435:00 +CPU= 21.962
16:20:31 STEP 2436 H=2436:00 +CPU= 39.485
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002436
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002436
16:20:58 STEP 2437 H=2437:00 +CPU= 26.500
16:21:24 STEP 2438 H=2438:00 +CPU= 26.267
16:21:57 STEP 2439 H=2439:00 +CPU= 33.345
16:22:18 STEP 2440 H=2440:00 +CPU= 21.043
16:22:39 STEP 2441 H=2441:00 +CPU= 20.645
16:23:16 STEP 2442 H=2442:00 +CPU= 37.415
16:23:41 STEP 2443 H=2443:00 +CPU= 24.696
16:24:04 STEP 2444 H=2444:00 +CPU= 23.066
16:24:44 STEP 2445 H=2445:00 +CPU= 39.396
16:25:12 STEP 2446 H=2446:00 +CPU= 28.842
16:25:43 STEP 2447 H=2447:00 +CPU= 30.394
16:26:22 STEP 2448 H=2448:00 +CPU= 38.207
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002448
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002424
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002436
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002436
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_45.zip
Uploading the intermediate file: upload_file_45.zip
16:26:42 STEP 2449 H=2449:00 +CPU= 20.342
Uploading trickle at timestep: 8812800
16:27:03 STEP 2450 H=2450:00 +CPU= 21.399
16:27:35 STEP 2451 H=2451:00 +CPU= 32.116
16:27:57 STEP 2452 H=2452:00 +CPU= 21.161
16:28:16 STEP 2453 H=2453:00 +CPU= 19.596
16:28:50 STEP 2454 H=2454:00 +CPU= 34.246
16:29:14 STEP 2455 H=2455:00 +CPU= 23.431
16:29:34 STEP 2456 H=2456:00 +CPU= 20.353
16:30:10 STEP 2457 H=2457:00 +CPU= 36.252
16:30:31 STEP 2458 H=2458:00 +CPU= 20.563
16:30:52 STEP 2459 H=2459:00 +CPU= 21.204
16:31:41 STEP 2460 H=2460:00 +CPU= 48.368
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002460
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002460
16:32:07 STEP 2461 H=2461:00 +CPU= 26.354
16:32:33 STEP 2462 H=2462:00 +CPU= 25.797
16:33:08 STEP 2463 H=2463:00 +CPU= 35.026
16:33:32 STEP 2464 H=2464:00 +CPU= 23.659
16:33:57 STEP 2465 H=2465:00 +CPU= 25.771
16:34:38 STEP 2466 H=2466:00 +CPU= 40.586
16:35:05 STEP 2467 H=2467:00 +CPU= 27.187
16:35:26 STEP 2468 H=2468:00 +CPU= 20.509
16:35:59 STEP 2469 H=2469:00 +CPU= 33.594
16:36:22 STEP 2470 H=2470:00 +CPU= 22.851
16:36:47 STEP 2471 H=2471:00 +CPU= 25.037
16:37:37 STEP 2472 H=2472:00 +CPU= 43.374
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002472
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002448
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002460
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002460
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_46.zip
Uploading the intermediate file: upload_file_46.zip
16:37:58 STEP 2473 H=2473:00 +CPU= 21.633
Uploading trickle at timestep: 8899200
16:38:21 STEP 2474 H=2474:00 +CPU= 22.204
16:39:01 STEP 2475 H=2475:00 +CPU= 40.778
16:39:25 STEP 2476 H=2476:00 +CPU= 24.033
16:39:48 STEP 2477 H=2477:00 +CPU= 22.976
16:40:30 STEP 2478 H=2478:00 +CPU= 41.965
16:40:52 STEP 2479 H=2479:00 +CPU= 21.438
16:41:20 STEP 2480 H=2480:00 +CPU= 28.528
16:42:00 STEP 2481 H=2481:00 +CPU= 39.306
16:42:21 STEP 2482 H=2482:00 +CPU= 21.023
16:42:41 STEP 2483 H=2483:00 +CPU= 19.881
16:43:17 STEP 2484 H=2484:00 +CPU= 36.347
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002484
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002484
16:43:38 STEP 2485 H=2485:00 +CPU= 21.245
16:44:00 STEP 2486 H=2486:00 +CPU= 21.548
16:44:39 STEP 2487 H=2487:00 +CPU= 39.014
16:45:02 STEP 2488 H=2488:00 +CPU= 22.786
16:45:21 STEP 2489 H=2489:00 +CPU= 19.326
16:45:55 STEP 2490 H=2490:00 +CPU= 33.846
16:46:16 STEP 2491 H=2491:00 +CPU= 20.813
16:46:38 STEP 2492 H=2492:00 +CPU= 22.576
16:47:18 STEP 2493 H=2493:00 +CPU= 39.361
16:47:40 STEP 2494 H=2494:00 +CPU= 22.013
16:48:02 STEP 2495 H=2495:00 +CPU= 22.092
16:48:52 STEP 2496 H=2496:00 +CPU= 48.206
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002496
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002472
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002484
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002484
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_47.zip
Uploading the intermediate file: upload_file_47.zip
Uploading trickle at timestep: 8985600
16:49:19 STEP 2497 H=2497:00 +CPU= 26.458
16:49:42 STEP 2498 H=2498:00 +CPU= 23.357
16:50:18 STEP 2499 H=2499:00 +CPU= 35.702
16:50:39 STEP 2500 H=2500:00 +CPU= 20.761
16:51:04 STEP 2501 H=2501:00 +CPU= 25.086
16:51:41 STEP 2502 H=2502:00 +CPU= 37.038
16:52:03 STEP 2503 H=2503:00 +CPU= 22.127
16:52:23 STEP 2504 H=2504:00 +CPU= 20.237
16:52:56 STEP 2505 H=2505:00 +CPU= 33.129
16:53:17 STEP 2506 H=2506:00 +CPU= 21.111
16:53:40 STEP 2507 H=2507:00 +CPU= 22.167
16:54:22 STEP 2508 H=2508:00 +CPU= 42.221
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002508
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002508
16:54:50 STEP 2509 H=2509:00 +CPU= 27.639
16:55:14 STEP 2510 H=2510:00 +CPU= 24.311
16:55:47 STEP 2511 H=2511:00 +CPU= 33.630
16:56:11 STEP 2512 H=2512:00 +CPU= 23.277
16:56:34 STEP 2513 H=2513:00 +CPU= 22.645
16:57:10 STEP 2514 H=2514:00 +CPU= 36.663
16:57:31 STEP 2515 H=2515:00 +CPU= 20.097
16:57:53 STEP 2516 H=2516:00 +CPU= 22.594
16:58:26 STEP 2517 H=2517:00 +CPU= 33.147
16:58:46 STEP 2518 H=2518:00 +CPU= 20.084
16:59:07 STEP 2519 H=2519:00 +CPU= 20.758
17:00:00 STEP 2520 H=2520:00 +CPU= 48.499
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002520
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002496
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002508
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002508
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_48.zip
Uploading the intermediate file: upload_file_48.zip
17:00:28 STEP 2521 H=2521:00 +CPU= 28.094
Uploading trickle at timestep: 9072000
17:00:57 STEP 2522 H=2522:00 +CPU= 28.651
17:01:31 STEP 2523 H=2523:00 +CPU= 34.208
17:01:53 STEP 2524 H=2524:00 +CPU= 22.254
17:02:13 STEP 2525 H=2525:00 +CPU= 19.210
17:02:43 STEP 2526 H=2526:00 +CPU= 30.909
17:03:05 STEP 2527 H=2527:00 +CPU= 21.646
17:03:26 STEP 2528 H=2528:00 +CPU= 20.406
17:03:57 STEP 2529 H=2529:00 +CPU= 31.980
17:04:21 STEP 2530 H=2530:00 +CPU= 23.051
17:04:40 STEP 2531 H=2531:00 +CPU= 19.367
17:05:14 STEP 2532 H=2532:00 +CPU= 33.934
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002532
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002532
17:05:33 STEP 2533 H=2533:00 +CPU= 19.182
17:05:55 STEP 2534 H=2534:00 +CPU= 21.497
17:06:36 STEP 2535 H=2535:00 +CPU= 41.800
17:07:04 STEP 2536 H=2536:00 +CPU= 27.957
17:07:28 STEP 2537 H=2537:00 +CPU= 23.539
17:08:05 STEP 2538 H=2538:00 +CPU= 36.886
17:08:24 STEP 2539 H=2539:00 +CPU= 19.394
17:08:46 STEP 2540 H=2540:00 +CPU= 21.860
17:09:17 STEP 2541 H=2541:00 +CPU= 31.026
17:09:37 STEP 2542 H=2542:00 +CPU= 20.194
17:10:02 STEP 2543 H=2543:00 +CPU= 24.450
17:10:51 STEP 2544 H=2544:00 +CPU= 48.888
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002544
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002520
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002532
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002532
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_49.zip
Uploading the intermediate file: upload_file_49.zip
Uploading trickle at timestep: 9158400
17:11:14 STEP 2545 H=2545:00 +CPU= 22.760
17:11:37 STEP 2546 H=2546:00 +CPU= 23.439
17:12:14 STEP 2547 H=2547:00 +CPU= 36.879
17:12:36 STEP 2548 H=2548:00 +CPU= 21.550
17:12:57 STEP 2549 H=2549:00 +CPU= 21.246
17:13:36 STEP 2550 H=2550:00 +CPU= 38.618
17:13:58 STEP 2551 H=2551:00 +CPU= 22.576
17:14:21 STEP 2552 H=2552:00 +CPU= 22.461
17:14:57 STEP 2553 H=2553:00 +CPU= 36.356
17:15:18 STEP 2554 H=2554:00 +CPU= 21.195
17:15:39 STEP 2555 H=2555:00 +CPU= 20.801
17:16:22 STEP 2556 H=2556:00 +CPU= 42.415
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002556
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002556
17:16:45 STEP 2557 H=2557:00 +CPU= 23.316
17:17:15 STEP 2558 H=2558:00 +CPU= 30.199
17:17:53 STEP 2559 H=2559:00 +CPU= 37.833
17:18:19 STEP 2560 H=2560:00 +CPU= 26.158
17:18:40 STEP 2561 H=2561:00 +CPU= 20.995
17:19:12 STEP 2562 H=2562:00 +CPU= 31.938
17:19:32 STEP 2563 H=2563:00 +CPU= 19.464
17:19:56 STEP 2564 H=2564:00 +CPU= 24.135
17:20:38 STEP 2565 H=2565:00 +CPU= 42.363
17:21:02 STEP 2566 H=2566:00 +CPU= 23.802
17:21:33 STEP 2567 H=2567:00 +CPU= 30.702
17:22:17 STEP 2568 H=2568:00 +CPU= 44.404
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002568
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002544
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002556
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002556
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_50.zip
Uploading the intermediate file: upload_file_50.zip
Uploading trickle at timestep: 9244800
17:22:45 STEP 2569 H=2569:00 +CPU= 27.457
17:23:11 STEP 2570 H=2570:00 +CPU= 26.030
17:23:45 STEP 2571 H=2571:00 +CPU= 33.598
17:24:07 STEP 2572 H=2572:00 +CPU= 22.525
17:24:28 STEP 2573 H=2573:00 +CPU= 21.311
17:25:04 STEP 2574 H=2574:00 +CPU= 35.838
17:25:30 STEP 2575 H=2575:00 +CPU= 26.014
17:25:55 STEP 2576 H=2576:00 +CPU= 25.136
17:26:41 STEP 2577 H=2577:00 +CPU= 45.883
17:27:05 STEP 2578 H=2578:00 +CPU= 23.234
17:27:25 STEP 2579 H=2579:00 +CPU= 19.821
17:28:01 STEP 2580 H=2580:00 +CPU= 36.788
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002580
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002580
17:28:22 STEP 2581 H=2581:00 +CPU= 20.785
17:28:43 STEP 2582 H=2582:00 +CPU= 20.537
17:29:17 STEP 2583 H=2583:00 +CPU= 34.168
17:29:37 STEP 2584 H=2584:00 +CPU= 20.444
17:29:58 STEP 2585 H=2585:00 +CPU= 20.681
17:30:32 STEP 2586 H=2586:00 +CPU= 34.550
17:30:55 STEP 2587 H=2587:00 +CPU= 22.359
17:31:18 STEP 2588 H=2588:00 +CPU= 22.628
17:31:55 STEP 2589 H=2589:00 +CPU= 37.652
17:32:19 STEP 2590 H=2590:00 +CPU= 23.936
17:32:44 STEP 2591 H=2591:00 +CPU= 24.347
17:33:21 STEP 2592 H=2592:00 +CPU= 37.771
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002592
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002592
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002568
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMGGhpi1+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMSHhpi1+002580
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248/ICMUAhpi1+002580
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036_51.zip
Uploading the intermediate file: upload_file_51.zip
17:33:43 STEP 2593 H=2593:00 +CPU= 21.716
Uploading trickle at timestep: 9331200
17:34:07 STEP 2594 H=2594:00 +CPU= 24.245
17:34:40 STEP 2595 H=2595:00 +CPU= 32.262
17:35:03 STEP 2596 H=2596:00 +CPU= 23.625
17:35:28 STEP 2597 H=2597:00 +CPU= 24.959
17:36:07 STEP 2598 H=2598:00 +CPU= 38.883
17:36:27 STEP 2599 H=2599:00 +CPU= 19.746
17:36:51 STEP 2600 H=2600:00 +CPU= 23.817
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.646] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 4035MB (heap), 4791MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.673] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.680] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.680] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.680] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 4035 MB (maxheap), 4791 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.680] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : SCAN2M
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL_HEAP
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : GP_MODEL
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADDRV
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADINTG
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : >OMP-RADINTG-RADLSW (1210)
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : RADIATION_SCHEME
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_interface:radiation
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173702:1669851422:34296.684] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : radiation_ifs_rrtm:gas_optics
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173703:1669851423:34297.692] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:18178:18178] [20221130:173703:1669851423:34297.692] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
master.exe 0000000001CE916B Unknown Unknown Unknown
master.exe 000000000081FFF1 Unknown Unknown Unknown
master.exe 0000000001DC9090 Unknown Unknown Unknown
master.exe 00000000018BC7B7 radiation_ifs_rrt 480 radiation_ifs_rrtm.F90
master.exe 0000000001862293 radiation_interfa 311 radiation_interface.F90
master.exe 00000000012D30E5 radiation_scheme_ 484 radiation_scheme.F90
master.exe 000000000129A415 radintg_ 1455 radintg.F90
master.exe 0000000001270124 raddrv_ 537 raddrv.F90
master.exe 0000000000BFD839 gp_model_ 482 gp_model.F90
master.exe 00000000012E9482 gp_model_heap_ 74 gp_model_heap.F90
master.exe 0000000000BBC1CE scan2m_ 535 scan2m.F90
master.exe 000000000057A664 stepo_ 327 stepo.F90
master.exe 000000000055DEEC cnt4_ 1133 cnt4.F90
master.exe 00000000005471C9 cnt3_ 267 cnt3.F90
master.exe 0000000000546412 cnt2_ 88 cnt2.F90
master.exe 0000000000545E28 cnt1_ 92 cnt1.F90
master.exe 000000000040708D cnt0_ 146 cnt0.F90
master.exe 000000000040220F MAIN__ 96 master.F90
master.exe 00000000004021A2 Unknown Unknown Unknown
master.exe 0000000001DCA390 Unknown Unknown Unknown
master.exe 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0159
(argv4) batchid: 945
(argv5) wuid: 12163248
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0eb7d3ab8996d74808da161df32d3399 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0159_2021050100_123_945_12163248.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0159_2021050100_123_945_12163248.zip
ic_ancil_file: ic_ancil_12163248
ifsdata_file: ifsdata_12163248
climate_data_file: clim_data_12163248
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b7c0ac4701eb9a0f5e8c67ad6d4af30d to: /var/lib/boinc-client/slots/6/ic_ancil_12163248.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12163248.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8f67a69fab2cc01af54da16e49e11755 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12163248.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12163248.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12163248.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12163248.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036
The child process has been launched with process id: 19590
Executing the command: /var/lib/boinc-client/slots/6/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:175634:1669852594:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
17:57:16 STEP 2592 H=2592:00 +CPU= 34.216
17:57:41 STEP 2593 H=2593:00 +CPU= 24.607
17:58:03 STEP 2594 H=2594:00 +CPU= 22.001
17:58:38 STEP 2595 H=2595:00 +CPU= 35.291
17:59:03 STEP 2596 H=2596:00 +CPU= 24.854
17:59:25 STEP 2597 H=2597:00 +CPU= 22.033
18:00:05 STEP 2598 H=2598:00 +CPU= 39.740
18:00:33 STEP 2599 H=2599:00 +CPU= 28.298
18:00:55 STEP 2600 H=2600:00 +CPU= 21.147
18:01:27 STEP 2601 H=2601:00 +CPU= 32.344
18:01:51 STEP 2602 H=2602:00 +CPU= 23.746
18:02:12 STEP 2603 H=2603:00 +CPU= 21.236
18:02:54 STEP 2604 H=2604:00 +CPU= 41.519
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMGGhpi1+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMSHhpi1+002604
Moving to projects directory: /var/lib/boinc-client/slots/6/ICMUAhpi1+002604
18:03:14 STEP 2605 H=2605:00 +CPU= 20.911
18:03:40 STEP 2606 H=2606:00 +CPU= 25.478
18:04:13 STEP 2607 H=2607:00 +CPU= 33.282
18:04:36 STEP 2608 H=2608:00 +CPU= 22.250
18:04:56 STEP 2609 H=2609:00 +CPU= 20.749
18:05:30 STEP 2610 H=2610:00 +CPU= 33.360
18:05:50 STEP 2611 H=2611:00 +CPU= 19.991
18:06:18 STEP 2612 H=2612:00 +CPU= 28.639
18:06:50 STEP 2613 H=2613:00 +CPU= 32.186
18:07:14 STEP 2614 H=2614:00 +CPU= 23.219
18:07:37 STEP 2615 H=2615:00 +CPU= 23.322
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.991] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#15 (SIGTERM) :: 2489MB (heap), 4788MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.991] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#15 was caused by unrecognized si_code [memaddr=0x1], nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#15, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 2572 MB (maxheap), 4788 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : STEPO
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IOPACK
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WRRESF
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IOSTREAM_MIX:IO_PUT
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180737:1669853257:662.992] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : IOSTREAM_MIX:WRITE_RECORD
double free or corruption (top)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180738:1669853258:663.997] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#15, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19590:19590] [20221130:180738:1669853258:663.997] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#15, nsigs = 1
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
master.exe 0000000001CE916B Unknown Unknown Unknown
master.exe 000000000081FFF1 Unknown Unknown Unknown
master.exe 0000000001DC9090 Unknown Unknown Unknown
master.exe 0000000001599114 Unknown Unknown Unknown
master.exe 00000000009D00E1 iostream_mix_mp_w 3188 iostream_mix.F90
master.exe 00000000009C2FBA iostream_mix_mp_i 1261 iostream_mix.F90
master.exe 00000000005F0878 wrresf_ 254 wrresf.F90
master.exe 00000000005DB0E2 iopack_ 196 iopack.F90
master.exe 0000000000579B82 stepo_ 253 stepo.F90
master.exe 000000000055D8D6 cnt4_ 923 cnt4.F90
master.exe 00000000005471C9 cnt3_ 267 cnt3.F90
master.exe 0000000000546412 cnt2_ 88 cnt2.F90
master.exe 0000000000545E28 cnt1_ 92 cnt1.F90
master.exe 000000000040708D cnt0_ 146 cnt0.F90
master.exe 000000000040220F MAIN__ 96 master.F90
master.exe 00000000004021A2 Unknown Unknown Unknown
master.exe 0000000001DCA390 Unknown Unknown Unknown
master.exe 000000000040206E Unknown Unknown Unknown
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.01_x86_64-pc-linux-gnu
(argv1) start_date: 2021050100
(argv2) exptid: hpi1
(argv3) unique_member_id: 0159
(argv4) batchid: 945
(argv5) wuid: 12163248
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/6
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.01
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12163248
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_app_1.01_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_0eb7d3ab8996d74808da161df32d3399 to: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0159_2021050100_123_945_12163248.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/6/oifs_43r3_ps_0159_2021050100_123_945_12163248.zip
ic_ancil_file: ic_ancil_12163248
ifsdata_file: ifsdata_12163248
climate_data_file: clim_data_12163248
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
Copying IC ancils from: ../../projects/climateprediction.net/jf_b7c0ac4701eb9a0f5e8c67ad6d4af30d to: /var/lib/boinc-client/slots/6/ic_ancil_12163248.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/6/ic_ancil_12163248.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_8f67a69fab2cc01af54da16e49e11755 to: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12163248.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/6/ifsdata/ifsdata_12163248.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/6/159l_2/clim_data_12163248.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/6/159l_2/clim_data_12163248.zip
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0159_2021050100_123_945_12163248_1_r1734682036
The child process has been launched with process id: 19834
Executing the command: /var/lib/boinc-client/slots/6/./master.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0 (RAND_MAX=2147483647)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192103:1669857663:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
ABORT! 1 !! *** WAVE MODEL HAS ABORTED
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3843] 1721 MB (maxheap), 3778 MB (maxrss), 0 MB (maxstack)
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.331] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : MASTER
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT0
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT2
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT3
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : CNT4
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WVCOUPLE
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : WAVEMDL
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : INITMDL
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.332] [c_drhook_print_@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:3897] : USERIN
SDL_TRACEBACK: Calling INTEL_TRBK, THRD = 1
Process 1 thread 1 calling linux_trbk from intel_trbk()
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:240] Backtrace(s) for program '/var/lib/boinc-client/slots/6/./master.exe' : sigcontextptr=0x7ffcb9e5fea0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:277] Backtrace (size = 19) with plain hex-dump
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [0]: [0x839273] : 0x839273
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [1]: [0x839879] : 0x839879
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [2]: [0x8c2957] : 0x8c2957
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [3]: [0x81d89e] : 0x81d89e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [4]: [0x15f8aca] : 0x15f8aca
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [5]: [0x15d51c8] : 0x15d51c8
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [6]: [0x166b07f] : 0x166b07f
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [7]: [0x15d7ff0] : 0x15d7ff0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [8]: [0x8711e0] : 0x8711e0
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [9]: [0x7e0165] : 0x7e0165
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [10]: [0x55e4ad] : 0x55e4ad
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [11]: [0x5471c9] : 0x5471c9
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [12]: [0x546412] : 0x546412
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [13]: [0x545e28] : 0x545e28
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [14]: [0x40708d] : 0x40708d
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [15]: [0x40220f] : 0x40220f
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [16]: [0x4021a2] : 0x4021a2
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [17]: [0x1dca390] : 0x1dca390
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:340] [18]: [0x40206e] : 0x40206e
[LinuxTraceBack@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/utilities/linuxtrbk.c:370] End of backtrace(s)
SDL_TRACEBACK: Done INTEL_TRBK, THRD = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1538] Received signal#6 (SIGABRT) :: 1721MB (heap), 3778MB (maxrss), 0MB (maxstack), 0 (paging), nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1542] Also activating Harakiri-alarm (SIGALRM=14) to expire after 500s elapsed to prevent hangs, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1544] Harakiri signal handler 'signal_harakiri' for signal#14 (SIGALRM) installed at 0x81f3c0 (old at (nil))
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1617] Signal#6 was caused by unrecognized si_code [memaddr=0x6b00004d7a], nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192139:1669857699:35.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1686] Starting DrHook backtrace for signal#6, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192140:1669857700:36.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1734] DrHook backtrace done for signal#6, nsigs = 1
[EC_DRHOOK:boinc-02.fullford.com:1:1:19834:19834] [20221130:192140:1669857700:36.373] [signal_drhook@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1785] Calling previous signal handler at 0x1ce8bb0 for signal#6, nsigs = 1
forrtl: error (76): Abort trap signal
Image PC Routine Line Source
master.exe 0000000001CE916B Unknown Unknown Unknown
master.exe 000000000081FFF1 Unknown Unknown Unknown
master.exe 0000000001DC9090 Unknown Unknown Unknown
master.exe 0000000001DC903B Unknown Unknown Unknown
master.exe 00000000008C2880 sdl_mod_mp_sdl_sr 100 sdl_mod.F90
master.exe 000000000081D8AF abor1_ 41 abor1.F90
master.exe 00000000015F8ACA mpabort_ 50 mpabort.F
master.exe 00000000015D51C8 abort1_ 44 abort1.F
master.exe 000000000166B07F userin_ 553 userin.F
master.exe 00000000015D7FF0 initmdl_ 452 initmdl.F
master.exe 00000000008711E0 wavemdl_ 393 wavemdl.F
master.exe 00000000007E0165 wvcouple_ 260 wvcouple.F90
master.exe 000000000055E4AD cnt4_ 1283 cnt4.F90
master.exe 00000000005471C9 cnt3_ 267 cnt3.F90
master.exe 0000000000546412 cnt2_ 88 cnt2.F90
master.exe 0000000000545E28 cnt1_ 92 cnt1.F90
master.exe 000000000040708D cnt0_ 146 cnt0.F90
master.exe 000000000040220F MAIN__ 96 master.F90
master.exe 00000000004021A2 Unknown Unknown Unknown
master.exe 0000000001DCA390 Unknown Unknown Unknown
master.exe 000000000040206E Unknown Unknown Unknown
The child process terminated with status: 0
..Failed, model did not complete successfully
</stderr_txt>
]]>
