climateprediction.net home page
Task 22265800

Task 22265800

Name oifs_43r3_ps_0167_1995050100_123_964_12180811_0
Workunit 12180811
Created 22 Dec 2022, 7:39:33 UTC
Sent 16 Jan 2023, 5:17:17 UTC
Report deadline 15 Feb 2023, 5:17:17 UTC
Received 16 Jan 2023, 21:30:35 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 1538037
Run time 14 hours 53 min 34 sec
CPU time 14 hours 53 min 34 sec
Validate state Invalid
Credit 8,709.12
Device peak FLOPS 4.76 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,777.81 MB
Peak swap size 4,978.08 MB
Peak disk usage 2,827.53 MB
Stderr
<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 9 (0x9, -247)</message>
<stderr_txt>
001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_55.zip
Uploading the intermediate file: upload_file_55.zip
  01:01:24 STEP 1345 H=1345:00 +CPU= 22.818
  01:01:47 STEP 1346 H=1346:00 +CPU= 23.104
  01:02:23 STEP 1347 H=1347:00 +CPU= 35.431
  01:02:46 STEP 1348 H=1348:00 +CPU= 23.303
  01:03:09 STEP 1349 H=1349:00 +CPU= 22.420
  01:03:46 STEP 1350 H=1350:00 +CPU= 36.980
  01:04:08 STEP 1351 H=1351:00 +CPU= 22.614
  01:04:32 STEP 1352 H=1352:00 +CPU= 23.661
  01:05:08 STEP 1353 H=1353:00 +CPU= 35.590
  01:05:31 STEP 1354 H=1354:00 +CPU= 23.235
  01:05:54 STEP 1355 H=1355:00 +CPU= 22.659
  01:06:33 STEP 1356 H=1356:00 +CPU= 39.332
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001356
  01:06:56 STEP 1357 H=1357:00 +CPU= 22.527
  01:07:19 STEP 1358 H=1358:00 +CPU= 23.247
  01:07:55 STEP 1359 H=1359:00 +CPU= 35.700
  01:08:18 STEP 1360 H=1360:00 +CPU= 23.076
  01:08:41 STEP 1361 H=1361:00 +CPU= 22.576
  01:09:18 STEP 1362 H=1362:00 +CPU= 36.437
  01:09:40 STEP 1363 H=1363:00 +CPU= 22.468
  01:10:03 STEP 1364 H=1364:00 +CPU= 22.825
  01:10:39 STEP 1365 H=1365:00 +CPU= 35.685
  01:11:02 STEP 1366 H=1366:00 +CPU= 23.188
  01:11:26 STEP 1367 H=1367:00 +CPU= 23.789
  01:12:08 STEP 1368 H=1368:00 +CPU= 42.072
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_56.zip
Uploading the intermediate file: upload_file_56.zip
  01:12:30 STEP 1369 H=1369:00 +CPU= 22.313
  01:12:53 STEP 1370 H=1370:00 +CPU= 23.141
  01:13:29 STEP 1371 H=1371:00 +CPU= 35.741
  01:13:52 STEP 1372 H=1372:00 +CPU= 22.970
  01:14:15 STEP 1373 H=1373:00 +CPU= 22.551
  01:14:51 STEP 1374 H=1374:00 +CPU= 36.178
  01:15:14 STEP 1375 H=1375:00 +CPU= 22.382
  01:15:37 STEP 1376 H=1376:00 +CPU= 22.948
  01:16:13 STEP 1377 H=1377:00 +CPU= 36.910
  01:16:37 STEP 1378 H=1378:00 +CPU= 23.426
  01:17:00 STEP 1379 H=1379:00 +CPU= 22.182
  01:17:39 STEP 1380 H=1380:00 +CPU= 39.750
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001380
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001380
  01:18:02 STEP 1381 H=1381:00 +CPU= 22.406
  01:18:25 STEP 1382 H=1382:00 +CPU= 23.176
  01:19:01 STEP 1383 H=1383:00 +CPU= 35.825
  01:19:24 STEP 1384 H=1384:00 +CPU= 23.401
  01:19:47 STEP 1385 H=1385:00 +CPU= 22.301
  01:20:23 STEP 1386 H=1386:00 +CPU= 36.671
  01:20:46 STEP 1387 H=1387:00 +CPU= 22.471
  01:21:10 STEP 1388 H=1388:00 +CPU= 23.234
  01:21:46 STEP 1389 H=1389:00 +CPU= 35.623
  01:22:11 STEP 1390 H=1390:00 +CPU= 23.264
  01:22:34 STEP 1391 H=1391:00 +CPU= 23.209
  01:23:17 STEP 1392 H=1392:00 +CPU= 43.133
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001392
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001380
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001380
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_57.zip
Uploading the intermediate file: upload_file_57.zip
  01:23:40 STEP 1393 H=1393:00 +CPU= 22.342
  01:24:03 STEP 1394 H=1394:00 +CPU= 23.008
  01:24:38 STEP 1395 H=1395:00 +CPU= 35.676
  01:25:01 STEP 1396 H=1396:00 +CPU= 23.167
  01:25:24 STEP 1397 H=1397:00 +CPU= 22.236
  01:26:00 STEP 1398 H=1398:00 +CPU= 35.978
  01:26:23 STEP 1399 H=1399:00 +CPU= 22.824
  01:26:47 STEP 1400 H=1400:00 +CPU= 22.918
  01:27:22 STEP 1401 H=1401:00 +CPU= 35.473
  01:27:45 STEP 1402 H=1402:00 +CPU= 22.976
  01:28:07 STEP 1403 H=1403:00 +CPU= 22.295
  01:28:47 STEP 1404 H=1404:00 +CPU= 39.301
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001404
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001404
  01:29:09 STEP 1405 H=1405:00 +CPU= 22.465
  01:29:33 STEP 1406 H=1406:00 +CPU= 23.008
  01:30:08 STEP 1407 H=1407:00 +CPU= 35.530
  01:30:32 STEP 1408 H=1408:00 +CPU= 23.232
  01:30:54 STEP 1409 H=1409:00 +CPU= 22.437
  01:31:31 STEP 1410 H=1410:00 +CPU= 36.527
  01:31:53 STEP 1411 H=1411:00 +CPU= 22.671
  01:32:16 STEP 1412 H=1412:00 +CPU= 23.015
  01:32:53 STEP 1413 H=1413:00 +CPU= 35.605
  01:33:16 STEP 1414 H=1414:00 +CPU= 23.481
  01:33:41 STEP 1415 H=1415:00 +CPU= 23.566
  01:34:27 STEP 1416 H=1416:00 +CPU= 42.552
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001416
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001392
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001404
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001404
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_58.zip
Uploading the intermediate file: upload_file_58.zip
  01:34:50 STEP 1417 H=1417:00 +CPU= 23.096
  01:35:13 STEP 1418 H=1418:00 +CPU= 23.189
  01:35:49 STEP 1419 H=1419:00 +CPU= 35.923
  01:36:13 STEP 1420 H=1420:00 +CPU= 23.477
  01:36:36 STEP 1421 H=1421:00 +CPU= 22.854
  01:37:12 STEP 1422 H=1422:00 +CPU= 36.400
  01:37:35 STEP 1423 H=1423:00 +CPU= 22.701
  01:37:58 STEP 1424 H=1424:00 +CPU= 23.105
  01:38:34 STEP 1425 H=1425:00 +CPU= 35.817
  01:38:57 STEP 1426 H=1426:00 +CPU= 23.262
  01:39:21 STEP 1427 H=1427:00 +CPU= 23.226
  01:40:01 STEP 1428 H=1428:00 +CPU= 40.170
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001428
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001428
  01:40:24 STEP 1429 H=1429:00 +CPU= 22.415
  01:40:47 STEP 1430 H=1430:00 +CPU= 23.043
  01:41:22 STEP 1431 H=1431:00 +CPU= 35.642
  01:41:46 STEP 1432 H=1432:00 +CPU= 23.656
  01:42:09 STEP 1433 H=1433:00 +CPU= 22.308
  01:42:45 STEP 1434 H=1434:00 +CPU= 36.774
  01:43:08 STEP 1435 H=1435:00 +CPU= 22.484
  01:43:31 STEP 1436 H=1436:00 +CPU= 23.021
  01:44:07 STEP 1437 H=1437:00 +CPU= 35.788
  01:44:30 STEP 1438 H=1438:00 +CPU= 23.119
  01:44:53 STEP 1439 H=1439:00 +CPU= 22.984
  01:45:38 STEP 1440 H=1440:00 +CPU= 41.750
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001440
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001416
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001428
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001428
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_59.zip
Uploading the intermediate file: upload_file_59.zip
  01:46:00 STEP 1441 H=1441:00 +CPU= 22.604
  01:46:24 STEP 1442 H=1442:00 +CPU= 22.900
  01:46:59 STEP 1443 H=1443:00 +CPU= 35.507
  01:47:22 STEP 1444 H=1444:00 +CPU= 23.172
  01:47:45 STEP 1445 H=1445:00 +CPU= 22.376
  01:48:21 STEP 1446 H=1446:00 +CPU= 36.187
  01:48:43 STEP 1447 H=1447:00 +CPU= 22.614
  01:49:06 STEP 1448 H=1448:00 +CPU= 23.032
  01:49:44 STEP 1449 H=1449:00 +CPU= 36.696
  01:50:08 STEP 1450 H=1450:00 +CPU= 23.178
  01:50:31 STEP 1451 H=1451:00 +CPU= 22.785
  01:51:11 STEP 1452 H=1452:00 +CPU= 39.439
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001452
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001452
  01:51:34 STEP 1453 H=1453:00 +CPU= 22.703
  01:51:57 STEP 1454 H=1454:00 +CPU= 23.238
  01:52:33 STEP 1455 H=1455:00 +CPU= 35.526
  01:52:57 STEP 1456 H=1456:00 +CPU= 23.718
  01:53:20 STEP 1457 H=1457:00 +CPU= 23.218
  01:53:58 STEP 1458 H=1458:00 +CPU= 37.230
  01:54:21 STEP 1459 H=1459:00 +CPU= 23.025
  01:54:44 STEP 1460 H=1460:00 +CPU= 23.384
  01:55:20 STEP 1461 H=1461:00 +CPU= 35.618
  01:55:43 STEP 1462 H=1462:00 +CPU= 23.066
  01:56:06 STEP 1463 H=1463:00 +CPU= 22.936
  01:56:48 STEP 1464 H=1464:00 +CPU= 41.822
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001464
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001440
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001452
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001452
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_60.zip
Uploading the intermediate file: upload_file_60.zip
  01:57:10 STEP 1465 H=1465:00 +CPU= 22.447
  01:57:34 STEP 1466 H=1466:00 +CPU= 23.372
  01:58:10 STEP 1467 H=1467:00 +CPU= 35.760
  01:58:34 STEP 1468 H=1468:00 +CPU= 23.086
  01:58:56 STEP 1469 H=1469:00 +CPU= 22.434
  01:59:34 STEP 1470 H=1470:00 +CPU= 37.185
  01:59:56 STEP 1471 H=1471:00 +CPU= 22.622
  02:00:20 STEP 1472 H=1472:00 +CPU= 23.264
  02:00:56 STEP 1473 H=1473:00 +CPU= 36.847
  02:01:20 STEP 1474 H=1474:00 +CPU= 23.158
  02:01:43 STEP 1475 H=1475:00 +CPU= 23.048
  02:02:23 STEP 1476 H=1476:00 +CPU= 40.114
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001476
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001476
  02:02:46 STEP 1477 H=1477:00 +CPU= 22.466
  02:03:09 STEP 1478 H=1478:00 +CPU= 23.205
  02:03:45 STEP 1479 H=1479:00 +CPU= 35.449
  02:04:08 STEP 1480 H=1480:00 +CPU= 23.124
  02:04:32 STEP 1481 H=1481:00 +CPU= 23.719
  02:05:10 STEP 1482 H=1482:00 +CPU= 37.741
  02:05:32 STEP 1483 H=1483:00 +CPU= 22.565
  02:05:55 STEP 1484 H=1484:00 +CPU= 23.066
  02:06:32 STEP 1485 H=1485:00 +CPU= 36.156
  02:06:55 STEP 1486 H=1486:00 +CPU= 23.218
  02:07:18 STEP 1487 H=1487:00 +CPU= 23.165
  02:08:00 STEP 1488 H=1488:00 +CPU= 42.035
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001488
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001464
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001476
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001476
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_61.zip
Uploading the intermediate file: upload_file_61.zip
  02:08:23 STEP 1489 H=1489:00 +CPU= 22.412
Uploading trickle at timestep: 5356800
  02:08:46 STEP 1490 H=1490:00 +CPU= 23.124
  02:09:22 STEP 1491 H=1491:00 +CPU= 36.116
  02:09:45 STEP 1492 H=1492:00 +CPU= 23.230
  02:10:08 STEP 1493 H=1493:00 +CPU= 22.669
  02:10:44 STEP 1494 H=1494:00 +CPU= 36.370
  02:11:07 STEP 1495 H=1495:00 +CPU= 22.547
  02:11:31 STEP 1496 H=1496:00 +CPU= 23.361
  02:12:07 STEP 1497 H=1497:00 +CPU= 35.882
  02:12:30 STEP 1498 H=1498:00 +CPU= 23.163
  02:12:53 STEP 1499 H=1499:00 +CPU= 22.336
  02:13:34 STEP 1500 H=1500:00 +CPU= 39.786
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001500
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001500
  02:13:56 STEP 1501 H=1501:00 +CPU= 22.495
  02:14:21 STEP 1502 H=1502:00 +CPU= 23.038
  02:14:56 STEP 1503 H=1503:00 +CPU= 35.469
  02:15:19 STEP 1504 H=1504:00 +CPU= 23.184
  02:15:43 STEP 1505 H=1505:00 +CPU= 23.202
  02:16:20 STEP 1506 H=1506:00 +CPU= 36.431
  02:16:42 STEP 1507 H=1507:00 +CPU= 22.504
  02:17:06 STEP 1508 H=1508:00 +CPU= 23.686
  02:17:42 STEP 1509 H=1509:00 +CPU= 36.019
  02:18:06 STEP 1510 H=1510:00 +CPU= 23.750
  02:18:29 STEP 1511 H=1511:00 +CPU= 23.222
  02:19:11 STEP 1512 H=1512:00 +CPU= 41.870
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001512
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001488
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001500
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001500
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_62.zip
Uploading the intermediate file: upload_file_62.zip
  02:19:33 STEP 1513 H=1513:00 +CPU= 22.344
  02:19:56 STEP 1514 H=1514:00 +CPU= 22.863
  02:20:32 STEP 1515 H=1515:00 +CPU= 35.500
  02:20:55 STEP 1516 H=1516:00 +CPU= 23.112
  02:21:17 STEP 1517 H=1517:00 +CPU= 22.333
  02:21:54 STEP 1518 H=1518:00 +CPU= 36.631
  02:22:16 STEP 1519 H=1519:00 +CPU= 22.689
  02:22:40 STEP 1520 H=1520:00 +CPU= 23.247
  02:23:18 STEP 1521 H=1521:00 +CPU= 36.954
  02:23:42 STEP 1522 H=1522:00 +CPU= 24.194
  02:24:04 STEP 1523 H=1523:00 +CPU= 22.571
  02:24:45 STEP 1524 H=1524:00 +CPU= 40.178
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001524
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001524
  02:25:08 STEP 1525 H=1525:00 +CPU= 22.888
  02:25:31 STEP 1526 H=1526:00 +CPU= 23.577
  02:26:08 STEP 1527 H=1527:00 +CPU= 35.920
  02:26:31 STEP 1528 H=1528:00 +CPU= 23.034
  02:26:54 STEP 1529 H=1529:00 +CPU= 22.374
  02:27:30 STEP 1530 H=1530:00 +CPU= 36.452
  02:27:53 STEP 1531 H=1531:00 +CPU= 22.414
  02:28:16 STEP 1532 H=1532:00 +CPU= 23.341
  02:28:52 STEP 1533 H=1533:00 +CPU= 35.498
  02:29:16 STEP 1534 H=1534:00 +CPU= 23.216
  02:29:39 STEP 1535 H=1535:00 +CPU= 23.119
  02:30:23 STEP 1536 H=1536:00 +CPU= 42.667
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001536
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001512
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001524
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001524
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_63.zip
Uploading the intermediate file: upload_file_63.zip
  02:30:46 STEP 1537 H=1537:00 +CPU= 22.791
  02:31:09 STEP 1538 H=1538:00 +CPU= 23.862
  02:31:45 STEP 1539 H=1539:00 +CPU= 35.898
  02:32:09 STEP 1540 H=1540:00 +CPU= 23.243
  02:32:31 STEP 1541 H=1541:00 +CPU= 22.577
  02:33:07 STEP 1542 H=1542:00 +CPU= 36.412
  02:33:30 STEP 1543 H=1543:00 +CPU= 22.535
  02:33:53 STEP 1544 H=1544:00 +CPU= 23.101
  02:34:30 STEP 1545 H=1545:00 +CPU= 35.837
  02:34:53 STEP 1546 H=1546:00 +CPU= 23.236
  02:35:16 STEP 1547 H=1547:00 +CPU= 22.301
  02:35:57 STEP 1548 H=1548:00 +CPU= 39.664
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001548
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001548
  02:36:19 STEP 1549 H=1549:00 +CPU= 22.737
  02:36:43 STEP 1550 H=1550:00 +CPU= 23.240
  02:37:19 STEP 1551 H=1551:00 +CPU= 35.848
  02:37:42 STEP 1552 H=1552:00 +CPU= 23.317
  02:38:05 STEP 1553 H=1553:00 +CPU= 22.370
  02:38:41 STEP 1554 H=1554:00 +CPU= 36.244
  02:39:04 STEP 1555 H=1555:00 +CPU= 23.127
  02:39:27 STEP 1556 H=1556:00 +CPU= 23.287
  02:40:03 STEP 1557 H=1557:00 +CPU= 35.625
  02:40:26 STEP 1558 H=1558:00 +CPU= 23.008
  02:40:49 STEP 1559 H=1559:00 +CPU= 23.372
  02:41:31 STEP 1560 H=1560:00 +CPU= 41.930
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001560
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001536
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001548
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001548
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_64.zip
Uploading the intermediate file: upload_file_64.zip
  02:41:54 STEP 1561 H=1561:00 +CPU= 22.511
  02:42:17 STEP 1562 H=1562:00 +CPU= 23.134
  02:42:54 STEP 1563 H=1563:00 +CPU= 36.406
  02:43:17 STEP 1564 H=1564:00 +CPU= 23.170
  02:43:40 STEP 1565 H=1565:00 +CPU= 22.493
  02:44:16 STEP 1566 H=1566:00 +CPU= 36.517
  02:44:39 STEP 1567 H=1567:00 +CPU= 22.398
  02:45:02 STEP 1568 H=1568:00 +CPU= 23.248
  02:45:37 STEP 1569 H=1569:00 +CPU= 35.593
  02:46:01 STEP 1570 H=1570:00 +CPU= 23.210
  02:46:23 STEP 1571 H=1571:00 +CPU= 22.373
  02:47:04 STEP 1572 H=1572:00 +CPU= 39.550
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001572
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001572
  02:47:27 STEP 1573 H=1573:00 +CPU= 22.556
  02:47:50 STEP 1574 H=1574:00 +CPU= 22.933
  02:48:27 STEP 1575 H=1575:00 +CPU= 35.494
  02:48:50 STEP 1576 H=1576:00 +CPU= 23.070
  02:49:12 STEP 1577 H=1577:00 +CPU= 22.346
  02:49:49 STEP 1578 H=1578:00 +CPU= 36.595
  02:50:11 STEP 1579 H=1579:00 +CPU= 22.394
  02:50:35 STEP 1580 H=1580:00 +CPU= 23.810
  02:51:12 STEP 1581 H=1581:00 +CPU= 36.349
  02:51:36 STEP 1582 H=1582:00 +CPU= 23.248
  02:52:01 STEP 1583 H=1583:00 +CPU= 23.853
  02:52:47 STEP 1584 H=1584:00 +CPU= 42.442
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001584
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001560
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001572
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001572
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_65.zip
Uploading the intermediate file: upload_file_65.zip
  02:53:10 STEP 1585 H=1585:00 +CPU= 22.519
  02:53:33 STEP 1586 H=1586:00 +CPU= 23.115
  02:54:08 STEP 1587 H=1587:00 +CPU= 35.707
  02:54:32 STEP 1588 H=1588:00 +CPU= 23.246
  02:54:55 STEP 1589 H=1589:00 +CPU= 22.498
  02:55:31 STEP 1590 H=1590:00 +CPU= 36.497
  02:55:54 STEP 1591 H=1591:00 +CPU= 22.532
  02:56:17 STEP 1592 H=1592:00 +CPU= 23.003
  02:56:52 STEP 1593 H=1593:00 +CPU= 35.670
  02:57:16 STEP 1594 H=1594:00 +CPU= 23.395
  02:57:38 STEP 1595 H=1595:00 +CPU= 22.722
  02:58:18 STEP 1596 H=1596:00 +CPU= 39.670
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001596
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001596
  02:58:42 STEP 1597 H=1597:00 +CPU= 23.466
  02:59:05 STEP 1598 H=1598:00 +CPU= 23.375
  02:59:41 STEP 1599 H=1599:00 +CPU= 35.713
  03:00:05 STEP 1600 H=1600:00 +CPU= 23.436
  03:00:27 STEP 1601 H=1601:00 +CPU= 22.445
  03:01:03 STEP 1602 H=1602:00 +CPU= 36.365
  03:01:26 STEP 1603 H=1603:00 +CPU= 22.629
  03:01:49 STEP 1604 H=1604:00 +CPU= 23.171
  03:02:25 STEP 1605 H=1605:00 +CPU= 35.484
  03:02:48 STEP 1606 H=1606:00 +CPU= 23.172
  03:03:15 STEP 1607 H=1607:00 +CPU= 23.154
  03:03:59 STEP 1608 H=1608:00 +CPU= 41.528
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001608
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001584
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001596
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001596
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_66.zip
Uploading the intermediate file: upload_file_66.zip
  03:04:23 STEP 1609 H=1609:00 +CPU= 22.387
  03:04:46 STEP 1610 H=1610:00 +CPU= 22.945
  03:05:22 STEP 1611 H=1611:00 +CPU= 35.465
  03:05:45 STEP 1612 H=1612:00 +CPU= 23.112
  03:06:08 STEP 1613 H=1613:00 +CPU= 23.006
  03:06:45 STEP 1614 H=1614:00 +CPU= 36.841
  03:07:08 STEP 1615 H=1615:00 +CPU= 22.500
  03:07:31 STEP 1616 H=1616:00 +CPU= 23.402
  03:08:07 STEP 1617 H=1617:00 +CPU= 35.963
  03:08:33 STEP 1618 H=1618:00 +CPU= 25.289
  03:08:55 STEP 1619 H=1619:00 +CPU= 22.632
  03:09:35 STEP 1620 H=1620:00 +CPU= 39.547
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001620
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001620
  03:09:58 STEP 1621 H=1621:00 +CPU= 22.530
  03:10:21 STEP 1622 H=1622:00 +CPU= 23.182
  03:10:58 STEP 1623 H=1623:00 +CPU= 35.218
  03:11:21 STEP 1624 H=1624:00 +CPU= 23.253
  03:11:44 STEP 1625 H=1625:00 +CPU= 22.512
  03:12:21 STEP 1626 H=1626:00 +CPU= 36.862
  03:12:44 STEP 1627 H=1627:00 +CPU= 22.559
  03:13:07 STEP 1628 H=1628:00 +CPU= 23.116
  03:13:43 STEP 1629 H=1629:00 +CPU= 36.229
  03:14:06 STEP 1630 H=1630:00 +CPU= 23.227
  03:14:33 STEP 1631 H=1631:00 +CPU= 23.388
  03:15:17 STEP 1632 H=1632:00 +CPU= 42.035
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001632
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001608
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001620
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001620
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_67.zip
Uploading the intermediate file: upload_file_67.zip
  03:15:40 STEP 1633 H=1633:00 +CPU= 22.859
  03:16:03 STEP 1634 H=1634:00 +CPU= 23.013
  03:16:39 STEP 1635 H=1635:00 +CPU= 35.485
  03:17:03 STEP 1636 H=1636:00 +CPU= 23.867
  03:17:26 STEP 1637 H=1637:00 +CPU= 22.618
  03:18:03 STEP 1638 H=1638:00 +CPU= 36.204
  03:18:27 STEP 1639 H=1639:00 +CPU= 23.222
  03:18:50 STEP 1640 H=1640:00 +CPU= 23.271
  03:19:26 STEP 1641 H=1641:00 +CPU= 35.538
  03:19:49 STEP 1642 H=1642:00 +CPU= 23.081
  03:20:11 STEP 1643 H=1643:00 +CPU= 22.415
  03:20:50 STEP 1644 H=1644:00 +CPU= 39.419
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001644
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001644
  03:21:13 STEP 1645 H=1645:00 +CPU= 22.482
  03:21:36 STEP 1646 H=1646:00 +CPU= 23.157
  03:22:12 STEP 1647 H=1647:00 +CPU= 35.687
  03:22:35 STEP 1648 H=1648:00 +CPU= 23.057
  03:22:58 STEP 1649 H=1649:00 +CPU= 22.503
  03:23:35 STEP 1650 H=1650:00 +CPU= 36.901
  03:23:58 STEP 1651 H=1651:00 +CPU= 22.523
  03:24:21 STEP 1652 H=1652:00 +CPU= 23.187
  03:24:56 STEP 1653 H=1653:00 +CPU= 35.211
  03:25:19 STEP 1654 H=1654:00 +CPU= 23.325
  03:25:43 STEP 1655 H=1655:00 +CPU= 23.252
  03:26:28 STEP 1656 H=1656:00 +CPU= 41.872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001656
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001632
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001644
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001644
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_68.zip
Uploading the intermediate file: upload_file_68.zip
  03:26:50 STEP 1657 H=1657:00 +CPU= 22.433
  03:27:13 STEP 1658 H=1658:00 +CPU= 22.974
  03:27:48 STEP 1659 H=1659:00 +CPU= 35.150
  03:28:13 STEP 1660 H=1660:00 +CPU= 23.967
  03:28:35 STEP 1661 H=1661:00 +CPU= 22.525
  03:29:12 STEP 1662 H=1662:00 +CPU= 36.373
  03:29:35 STEP 1663 H=1663:00 +CPU= 22.613
  03:29:59 STEP 1664 H=1664:00 +CPU= 23.549
  03:30:35 STEP 1665 H=1665:00 +CPU= 36.012
  03:30:59 STEP 1666 H=1666:00 +CPU= 22.980
  03:31:21 STEP 1667 H=1667:00 +CPU= 22.734
  03:32:02 STEP 1668 H=1668:00 +CPU= 39.935
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001668
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001668
  03:32:25 STEP 1669 H=1669:00 +CPU= 22.456
  03:32:48 STEP 1670 H=1670:00 +CPU= 23.129
  03:33:23 STEP 1671 H=1671:00 +CPU= 35.365
  03:33:47 STEP 1672 H=1672:00 +CPU= 23.111
  03:34:10 STEP 1673 H=1673:00 +CPU= 22.500
  03:34:47 STEP 1674 H=1674:00 +CPU= 37.528
  03:35:10 STEP 1675 H=1675:00 +CPU= 22.546
  03:35:33 STEP 1676 H=1676:00 +CPU= 22.976
  03:36:09 STEP 1677 H=1677:00 +CPU= 35.843
  03:36:34 STEP 1678 H=1678:00 +CPU= 23.081
  03:37:00 STEP 1679 H=1679:00 +CPU= 23.114
  03:37:47 STEP 1680 H=1680:00 +CPU= 41.919
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001680
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001656
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001668
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001668
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_69.zip
Uploading the intermediate file: upload_file_69.zip
  03:38:10 STEP 1681 H=1681:00 +CPU= 22.404
  03:38:33 STEP 1682 H=1682:00 +CPU= 23.098
  03:39:08 STEP 1683 H=1683:00 +CPU= 35.452
  03:39:31 STEP 1684 H=1684:00 +CPU= 23.012
  03:39:55 STEP 1685 H=1685:00 +CPU= 22.451
  03:40:31 STEP 1686 H=1686:00 +CPU= 36.247
  03:40:55 STEP 1687 H=1687:00 +CPU= 22.658
  03:41:18 STEP 1688 H=1688:00 +CPU= 23.200
  03:41:54 STEP 1689 H=1689:00 +CPU= 35.396
  03:42:18 STEP 1690 H=1690:00 +CPU= 23.157
  03:42:40 STEP 1691 H=1691:00 +CPU= 22.515
  03:43:23 STEP 1692 H=1692:00 +CPU= 40.050
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001692
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001692
  03:43:46 STEP 1693 H=1693:00 +CPU= 22.550
  03:44:09 STEP 1694 H=1694:00 +CPU= 23.359
  03:44:45 STEP 1695 H=1695:00 +CPU= 35.292
  03:45:08 STEP 1696 H=1696:00 +CPU= 22.888
  03:45:30 STEP 1697 H=1697:00 +CPU= 22.279
  03:46:06 STEP 1698 H=1698:00 +CPU= 36.411
  03:46:30 STEP 1699 H=1699:00 +CPU= 23.365
  03:46:53 STEP 1700 H=1700:00 +CPU= 23.089
  03:47:28 STEP 1701 H=1701:00 +CPU= 35.400
  03:47:52 STEP 1702 H=1702:00 +CPU= 22.757
  03:48:16 STEP 1703 H=1703:00 +CPU= 23.607
  03:49:01 STEP 1704 H=1704:00 +CPU= 43.729
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001704
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001680
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001692
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001692
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_70.zip
Uploading the intermediate file: upload_file_70.zip
  03:49:24 STEP 1705 H=1705:00 +CPU= 23.228
  03:49:47 STEP 1706 H=1706:00 +CPU= 23.532
  03:50:23 STEP 1707 H=1707:00 +CPU= 35.406
  03:50:47 STEP 1708 H=1708:00 +CPU= 23.797
  03:51:10 STEP 1709 H=1709:00 +CPU= 22.744
  03:51:47 STEP 1710 H=1710:00 +CPU= 36.741
  03:52:10 STEP 1711 H=1711:00 +CPU= 22.965
  03:52:33 STEP 1712 H=1712:00 +CPU= 23.362
  03:53:10 STEP 1713 H=1713:00 +CPU= 36.144
  03:53:34 STEP 1714 H=1714:00 +CPU= 23.733
  03:53:57 STEP 1715 H=1715:00 +CPU= 22.636
  03:54:36 STEP 1716 H=1716:00 +CPU= 38.920
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001716
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001716
  03:54:58 STEP 1717 H=1717:00 +CPU= 22.237
  03:55:22 STEP 1718 H=1718:00 +CPU= 23.522
  03:55:58 STEP 1719 H=1719:00 +CPU= 36.045
  03:56:22 STEP 1720 H=1720:00 +CPU= 23.435
  03:56:44 STEP 1721 H=1721:00 +CPU= 22.476
  03:57:20 STEP 1722 H=1722:00 +CPU= 36.366
  03:57:43 STEP 1723 H=1723:00 +CPU= 22.324
  03:58:06 STEP 1724 H=1724:00 +CPU= 23.067
  03:58:41 STEP 1725 H=1725:00 +CPU= 35.235
  03:59:04 STEP 1726 H=1726:00 +CPU= 22.944
  03:59:28 STEP 1727 H=1727:00 +CPU= 23.572
  04:00:12 STEP 1728 H=1728:00 +CPU= 41.589
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001728
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001704
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001716
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001716
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_71.zip
Uploading the intermediate file: upload_file_71.zip
  04:00:35 STEP 1729 H=1729:00 +CPU= 22.368
Uploading trickle at timestep: 6220800
  04:00:58 STEP 1730 H=1730:00 +CPU= 22.776
  04:01:33 STEP 1731 H=1731:00 +CPU= 35.039
  04:01:57 STEP 1732 H=1732:00 +CPU= 23.894
  04:02:19 STEP 1733 H=1733:00 +CPU= 22.758
  04:02:56 STEP 1734 H=1734:00 +CPU= 36.548
  04:03:18 STEP 1735 H=1735:00 +CPU= 22.496
  04:03:42 STEP 1736 H=1736:00 +CPU= 23.288
  04:04:17 STEP 1737 H=1737:00 +CPU= 35.565
  04:04:40 STEP 1738 H=1738:00 +CPU= 23.043
  04:05:03 STEP 1739 H=1739:00 +CPU= 23.050
  04:05:43 STEP 1740 H=1740:00 +CPU= 39.781
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001740
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001740
  04:06:06 STEP 1741 H=1741:00 +CPU= 22.463
  04:06:29 STEP 1742 H=1742:00 +CPU= 22.826
  04:07:06 STEP 1743 H=1743:00 +CPU= 35.571
  04:07:29 STEP 1744 H=1744:00 +CPU= 23.205
  04:07:52 STEP 1745 H=1745:00 +CPU= 22.537
  04:08:29 STEP 1746 H=1746:00 +CPU= 36.631
  04:08:51 STEP 1747 H=1747:00 +CPU= 22.455
  04:09:14 STEP 1748 H=1748:00 +CPU= 22.992
  04:09:49 STEP 1749 H=1749:00 +CPU= 35.283
  04:10:12 STEP 1750 H=1750:00 +CPU= 23.012
  04:10:35 STEP 1751 H=1751:00 +CPU= 23.058
  04:11:18 STEP 1752 H=1752:00 +CPU= 42.331
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001752
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001728
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001740
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001740
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_72.zip
Uploading the intermediate file: upload_file_72.zip
  04:11:40 STEP 1753 H=1753:00 +CPU= 22.570
  04:12:03 STEP 1754 H=1754:00 +CPU= 23.046
  04:12:39 STEP 1755 H=1755:00 +CPU= 35.535
  04:13:02 STEP 1756 H=1756:00 +CPU= 23.215
  04:13:25 STEP 1757 H=1757:00 +CPU= 22.752
  04:14:02 STEP 1758 H=1758:00 +CPU= 36.728
  04:14:24 STEP 1759 H=1759:00 +CPU= 22.480
  04:14:48 STEP 1760 H=1760:00 +CPU= 23.581
  04:15:23 STEP 1761 H=1761:00 +CPU= 35.557
  04:15:47 STEP 1762 H=1762:00 +CPU= 23.272
  04:16:10 STEP 1763 H=1763:00 +CPU= 23.377
  04:16:49 STEP 1764 H=1764:00 +CPU= 39.266
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001764
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001764
  04:17:12 STEP 1765 H=1765:00 +CPU= 22.310
  04:17:35 STEP 1766 H=1766:00 +CPU= 23.266
  04:18:11 STEP 1767 H=1767:00 +CPU= 35.431
  04:18:34 STEP 1768 H=1768:00 +CPU= 23.577
  04:18:58 STEP 1769 H=1769:00 +CPU= 23.073
  04:19:35 STEP 1770 H=1770:00 +CPU= 37.004
  04:19:57 STEP 1771 H=1771:00 +CPU= 22.468
  04:20:21 STEP 1772 H=1772:00 +CPU= 23.363
  04:20:56 STEP 1773 H=1773:00 +CPU= 35.505
  04:21:19 STEP 1774 H=1774:00 +CPU= 23.061
  04:21:43 STEP 1775 H=1775:00 +CPU= 23.158
  04:22:24 STEP 1776 H=1776:00 +CPU= 41.441
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001776
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001752
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001764
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001764
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_73.zip
Uploading the intermediate file: upload_file_73.zip
  04:22:46 STEP 1777 H=1777:00 +CPU= 22.351
  04:23:10 STEP 1778 H=1778:00 +CPU= 23.098
  04:23:45 STEP 1779 H=1779:00 +CPU= 35.366
  04:24:08 STEP 1780 H=1780:00 +CPU= 23.143
  04:24:31 STEP 1781 H=1781:00 +CPU= 22.765
  04:25:08 STEP 1782 H=1782:00 +CPU= 36.297
  04:25:30 STEP 1783 H=1783:00 +CPU= 22.418
  04:25:53 STEP 1784 H=1784:00 +CPU= 23.117
  04:26:29 STEP 1785 H=1785:00 +CPU= 35.457
  04:26:52 STEP 1786 H=1786:00 +CPU= 23.096
  04:27:14 STEP 1787 H=1787:00 +CPU= 22.478
  04:27:54 STEP 1788 H=1788:00 +CPU= 39.248
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001788
  04:28:16 STEP 1789 H=1789:00 +CPU= 22.515
  04:28:39 STEP 1790 H=1790:00 +CPU= 22.955
  04:29:15 STEP 1791 H=1791:00 +CPU= 35.249
  04:29:38 STEP 1792 H=1792:00 +CPU= 23.262
  04:30:01 STEP 1793 H=1793:00 +CPU= 22.311
  04:30:36 STEP 1794 H=1794:00 +CPU= 35.278
  04:30:58 STEP 1795 H=1795:00 +CPU= 21.864
  04:31:21 STEP 1796 H=1796:00 +CPU= 22.375
  04:31:55 STEP 1797 H=1797:00 +CPU= 34.426
  04:32:18 STEP 1798 H=1798:00 +CPU= 22.625
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 1995050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0167
(argv4) batchid: 964
(argv5) wuid: 12180811
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/19
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_576cd11d018f288f8415ef3da52d6747 to: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0167_1995050100_123_964_12180811.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/19/oifs_43r3_ps_0167_1995050100_123_964_12180811.zip
ic_ancil_file: ic_ancil_12180811
ifsdata_file: ifsdata_12180811
climate_data_file: clim_data_12180811
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_28f1a768107ee09a61dea5b8b3b5b90e to: /var/lib/boinc-client/slots/19/ic_ancil_12180811.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/19/ic_ancil_12180811.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_04b226e07484d59e5f7b2cff544e98cb to: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12180811.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/19/ifsdata/ifsdata_12180811.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/19/159l_2/clim_data_12180811.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/19/159l_2/clim_data_12180811.zip
Checking for progress XML file: /var/lib/boinc-client/slots/19/progress_file_12180811.xml
Opened progress file ok : /var/lib/boinc-client/slots/19/progress_file_12180811.xml
-- Model is restarting --
Adjusting last_iter, 1799, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/19/progress_file_12180811.xml
last_cpu_time: 50605
upload_file_number: 74
last_iter: 1775
last_upload: 6393600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927
The child process has been launched with process id: 433028
Executing the command: /var/lib/boinc-client/slots/19/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:debian:1:1:433028:433028] [20230117:052655:1673900815:0.001] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  05:27:46 STEP 1776 H=1776:00 +CPU= 45.888
  05:28:14 STEP 1777 H=1777:00 +CPU= 24.966
  05:28:40 STEP 1778 H=1778:00 +CPU= 25.534
  05:29:21 STEP 1779 H=1779:00 +CPU= 38.135
  05:29:47 STEP 1780 H=1780:00 +CPU= 25.865
  05:30:13 STEP 1781 H=1781:00 +CPU= 25.447
  05:30:54 STEP 1782 H=1782:00 +CPU= 40.725
  05:31:19 STEP 1783 H=1783:00 +CPU= 24.830
  05:31:45 STEP 1784 H=1784:00 +CPU= 25.959
  05:32:23 STEP 1785 H=1785:00 +CPU= 38.393
  05:32:48 STEP 1786 H=1786:00 +CPU= 25.202
  05:33:13 STEP 1787 H=1787:00 +CPU= 24.872
  05:33:56 STEP 1788 H=1788:00 +CPU= 42.616
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001788
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001788
  05:34:20 STEP 1789 H=1789:00 +CPU= 24.039
  05:34:45 STEP 1790 H=1790:00 +CPU= 25.163
  05:35:24 STEP 1791 H=1791:00 +CPU= 38.801
  05:35:49 STEP 1792 H=1792:00 +CPU= 25.191
  05:36:14 STEP 1793 H=1793:00 +CPU= 24.504
  05:36:56 STEP 1794 H=1794:00 +CPU= 38.958
  05:37:21 STEP 1795 H=1795:00 +CPU= 25.069
  05:37:47 STEP 1796 H=1796:00 +CPU= 25.087
  05:38:26 STEP 1797 H=1797:00 +CPU= 39.237
  05:38:51 STEP 1798 H=1798:00 +CPU= 25.260
  05:39:24 STEP 1799 H=1799:00 +CPU= 25.674
  05:40:17 STEP 1800 H=1800:00 +CPU= 45.436
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001800
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001776
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001788
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001788
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_74.zip
Uploading the intermediate file: upload_file_74.zip
  05:40:41 STEP 1801 H=1801:00 +CPU= 24.203
  05:41:07 STEP 1802 H=1802:00 +CPU= 25.470
  05:41:52 STEP 1803 H=1803:00 +CPU= 38.477
  05:42:18 STEP 1804 H=1804:00 +CPU= 25.622
  05:42:43 STEP 1805 H=1805:00 +CPU= 25.059
  05:43:23 STEP 1806 H=1806:00 +CPU= 40.586
  05:43:49 STEP 1807 H=1807:00 +CPU= 25.476
  05:44:15 STEP 1808 H=1808:00 +CPU= 25.862
  05:44:55 STEP 1809 H=1809:00 +CPU= 38.904
  05:45:20 STEP 1810 H=1810:00 +CPU= 25.404
  05:45:50 STEP 1811 H=1811:00 +CPU= 24.983
  05:46:32 STEP 1812 H=1812:00 +CPU= 42.159
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001812
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001812
  05:46:56 STEP 1813 H=1813:00 +CPU= 24.093
  05:47:22 STEP 1814 H=1814:00 +CPU= 25.971
  05:48:01 STEP 1815 H=1815:00 +CPU= 38.683
  05:48:26 STEP 1816 H=1816:00 +CPU= 24.965
  05:48:51 STEP 1817 H=1817:00 +CPU= 24.603
  05:49:31 STEP 1818 H=1818:00 +CPU= 40.028
  05:49:56 STEP 1819 H=1819:00 +CPU= 25.197
  05:50:21 STEP 1820 H=1820:00 +CPU= 25.349
  05:51:01 STEP 1821 H=1821:00 +CPU= 40.120
  05:51:27 STEP 1822 H=1822:00 +CPU= 25.749
  05:51:55 STEP 1823 H=1823:00 +CPU= 25.316
  05:52:45 STEP 1824 H=1824:00 +CPU= 46.524
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001824
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001800
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001812
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001812
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_75.zip
Uploading the intermediate file: upload_file_75.zip
  05:53:10 STEP 1825 H=1825:00 +CPU= 24.851
  05:53:36 STEP 1826 H=1826:00 +CPU= 25.787
  05:54:15 STEP 1827 H=1827:00 +CPU= 38.420
  05:54:40 STEP 1828 H=1828:00 +CPU= 25.082
  05:55:05 STEP 1829 H=1829:00 +CPU= 24.626
  05:55:44 STEP 1830 H=1830:00 +CPU= 39.819
  05:56:10 STEP 1831 H=1831:00 +CPU= 25.541
  05:56:36 STEP 1832 H=1832:00 +CPU= 26.252
  05:57:16 STEP 1833 H=1833:00 +CPU= 39.904
  05:57:41 STEP 1834 H=1834:00 +CPU= 25.215
  05:58:06 STEP 1835 H=1835:00 +CPU= 24.485
  05:58:48 STEP 1836 H=1836:00 +CPU= 42.460
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001836
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001836
  05:59:13 STEP 1837 H=1837:00 +CPU= 24.779
  05:59:38 STEP 1838 H=1838:00 +CPU= 25.142
  06:00:18 STEP 1839 H=1839:00 +CPU= 39.187
  06:00:44 STEP 1840 H=1840:00 +CPU= 25.898
  06:01:09 STEP 1841 H=1841:00 +CPU= 25.142
  06:01:49 STEP 1842 H=1842:00 +CPU= 39.657
  06:02:14 STEP 1843 H=1843:00 +CPU= 24.629
  06:02:40 STEP 1844 H=1844:00 +CPU= 25.446
  06:03:18 STEP 1845 H=1845:00 +CPU= 38.759
  06:03:44 STEP 1846 H=1846:00 +CPU= 25.253
  06:04:09 STEP 1847 H=1847:00 +CPU= 25.265
  06:04:58 STEP 1848 H=1848:00 +CPU= 47.092
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001848
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001824
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001836
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001836
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_76.zip
Uploading the intermediate file: upload_file_76.zip
  06:05:24 STEP 1849 H=1849:00 +CPU= 24.420
  06:05:49 STEP 1850 H=1850:00 +CPU= 24.863
  06:06:28 STEP 1851 H=1851:00 +CPU= 39.051
  06:06:53 STEP 1852 H=1852:00 +CPU= 25.354
  06:07:18 STEP 1853 H=1853:00 +CPU= 24.671
  06:07:57 STEP 1854 H=1854:00 +CPU= 39.743
  06:08:22 STEP 1855 H=1855:00 +CPU= 24.553
  06:08:47 STEP 1856 H=1856:00 +CPU= 25.376
  06:09:27 STEP 1857 H=1857:00 +CPU= 39.128
  06:09:52 STEP 1858 H=1858:00 +CPU= 25.119
  06:10:16 STEP 1859 H=1859:00 +CPU= 24.574
  06:11:00 STEP 1860 H=1860:00 +CPU= 43.043
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001860
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001860
  06:11:25 STEP 1861 H=1861:00 +CPU= 24.384
  06:11:50 STEP 1862 H=1862:00 +CPU= 25.214
  06:12:29 STEP 1863 H=1863:00 +CPU= 38.482
  06:12:55 STEP 1864 H=1864:00 +CPU= 25.777
  06:13:20 STEP 1865 H=1865:00 +CPU= 24.745
  06:13:59 STEP 1866 H=1866:00 +CPU= 38.971
  06:14:24 STEP 1867 H=1867:00 +CPU= 24.925
  06:14:49 STEP 1868 H=1868:00 +CPU= 25.015
  06:15:28 STEP 1869 H=1869:00 +CPU= 38.871
  06:15:53 STEP 1870 H=1870:00 +CPU= 25.292
  06:16:19 STEP 1871 H=1871:00 +CPU= 25.120
  06:17:07 STEP 1872 H=1872:00 +CPU= 45.924
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMGGhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMSHhq0f+001872
Moving to projects directory: /var/lib/boinc-client/slots/19/ICMUAhq0f+001872
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001848
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMGGhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMSHhq0f+001860
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12180811/ICMUAhq0f+001860
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0167_1995050100_123_964_12180811_0_r1207195927_77.zip
Uploading the intermediate file: upload_file_77.zip
  06:17:32 STEP 1873 H=1873:00 +CPU= 24.869
  06:17:57 STEP 1874 H=1874:00 +CPU= 25.170
double free or corruption (out)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:22:49 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 6,220,800 48,676 0.0078
09 May 2023 16:22:41 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 5,356,800 42,020 0.0078
09 May 2023 16:22:32 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 4,492,800 35,359 0.0079
09 May 2023 16:22:28 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 3,628,800 28,708 0.0079
09 May 2023 16:22:28 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 3,628,800 28,708 0.0079
09 May 2023 16:22:15 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 2,592,000 20,443 0.0079
09 May 2023 16:22:08 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 1,728,000 13,706 0.0079
09 May 2023 16:22:04 1538037 22265800 oifs_43r3_ps_0167_1995050100_123_964_12180811_0 864,000 6,969 0.0081


©2024 climateprediction.net