climateprediction.net home page
Task 22285731

Task 22285731

Name oifs_43r3_ps_0900_2013050100_123_982_12199544_0
Workunit 12199544
Created 24 Dec 2022, 8:46:20 UTC
Sent 23 Jan 2023, 0:33:11 UTC
Report deadline 24 Mar 2023, 0:33:11 UTC
Received 15 Feb 2023, 0:30:16 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 196 (0x000000C4) EXIT_DISK_LIMIT_EXCEEDED
Computer ID 1534673
Run time 12 hours 34 min 12 sec
CPU time 34 min 53 sec
Validate state Invalid
Credit 6,048.00
Device peak FLOPS 4.67 GFLOPS
Application version OpenIFS 43r3 Perturbed Surface v1.05
x86_64-pc-linux-gnu
Peak working set size 4,416.63 MB
Peak swap size 4,978.07 MB
Peak disk usage 7,178.75 MB
Stderr
<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
Disk usage limit exceeded</message>
<stderr_txt>
urong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:322142:322142] [20230214:210103:1676379663:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  21:01:47 STEP  960 H= 960:00 +CPU= 41.337
  21:02:08 STEP  961 H= 961:00 +CPU= 21.288
  21:02:30 STEP  962 H= 962:00 +CPU= 21.746
  21:03:04 STEP  963 H= 963:00 +CPU= 34.506
  21:03:26 STEP  964 H= 964:00 +CPU= 21.314
  21:03:46 STEP  965 H= 965:00 +CPU= 20.573
  21:04:21 STEP  966 H= 966:00 +CPU= 34.601
  21:04:41 STEP  967 H= 967:00 +CPU= 20.635
  21:05:03 STEP  968 H= 968:00 +CPU= 21.162
  21:05:36 STEP  969 H= 969:00 +CPU= 33.056
  21:05:57 STEP  970 H= 970:00 +CPU= 21.217
  21:06:17 STEP  971 H= 971:00 +CPU= 20.605
  21:06:56 STEP  972 H= 972:00 +CPU= 38.134
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000972
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000972
  21:07:18 STEP  973 H= 973:00 +CPU= 22.737
  21:07:42 STEP  974 H= 974:00 +CPU= 23.521
  21:08:19 STEP  975 H= 975:00 +CPU= 36.823
  21:08:41 STEP  976 H= 976:00 +CPU= 22.550
  21:09:02 STEP  977 H= 977:00 +CPU= 21.196
  21:09:38 STEP  978 H= 978:00 +CPU= 35.692
  21:09:59 STEP  979 H= 979:00 +CPU= 20.904
  21:10:21 STEP  980 H= 980:00 +CPU= 21.816
  21:10:54 STEP  981 H= 981:00 +CPU= 33.333
  21:11:15 STEP  982 H= 982:00 +CPU= 21.179
  21:11:37 STEP  983 H= 983:00 +CPU= 21.065
  21:12:16 STEP  984 H= 984:00 +CPU= 39.180
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+000960
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+000972
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+000972
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_40.zip
Uploading the intermediate file: upload_file_40.zip
  21:12:36 STEP  985 H= 985:00 +CPU= 20.578
  21:12:58 STEP  986 H= 986:00 +CPU= 21.351
  21:13:32 STEP  987 H= 987:00 +CPU= 34.163
  21:13:53 STEP  988 H= 988:00 +CPU= 21.400
  21:14:14 STEP  989 H= 989:00 +CPU= 20.499
  21:14:48 STEP  990 H= 990:00 +CPU= 33.893
  21:15:08 STEP  991 H= 991:00 +CPU= 20.772
  21:15:32 STEP  992 H= 992:00 +CPU= 23.272
  21:16:05 STEP  993 H= 993:00 +CPU= 33.729
  21:16:27 STEP  994 H= 994:00 +CPU= 21.713
  21:16:48 STEP  995 H= 995:00 +CPU= 20.958
  21:17:26 STEP  996 H= 996:00 +CPU= 38.243
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+000996
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+000996
  21:17:48 STEP  997 H= 997:00 +CPU= 21.630
  21:18:09 STEP  998 H= 998:00 +CPU= 21.288
  21:18:43 STEP  999 H= 999:00 +CPU= 33.879
  21:19:05 STEP 1000 H=1000:00 +CPU= 21.471
  21:19:27 STEP 1001 H=1001:00 +CPU= 22.210
  21:20:01 STEP 1002 H=1002:00 +CPU= 34.100
  21:20:23 STEP 1003 H=1003:00 +CPU= 21.567
  21:20:45 STEP 1004 H=1004:00 +CPU= 22.146
  21:21:20 STEP 1005 H=1005:00 +CPU= 35.018
  21:21:42 STEP 1006 H=1006:00 +CPU= 22.215
  21:22:04 STEP 1007 H=1007:00 +CPU= 21.875
  21:22:43 STEP 1008 H=1008:00 +CPU= 39.172
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001008
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+000984
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+000996
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+000996
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_41.zip
Uploading the intermediate file: upload_file_41.zip
  21:23:04 STEP 1009 H=1009:00 +CPU= 21.312
  21:23:26 STEP 1010 H=1010:00 +CPU= 21.356
  21:23:59 STEP 1011 H=1011:00 +CPU= 33.241
  21:24:21 STEP 1012 H=1012:00 +CPU= 21.538
  21:24:41 STEP 1013 H=1013:00 +CPU= 20.698
  21:25:15 STEP 1014 H=1014:00 +CPU= 33.877
  21:25:36 STEP 1015 H=1015:00 +CPU= 20.643
  21:25:57 STEP 1016 H=1016:00 +CPU= 21.406
  21:26:32 STEP 1017 H=1017:00 +CPU= 34.301
  21:26:53 STEP 1018 H=1018:00 +CPU= 21.793
  21:27:15 STEP 1019 H=1019:00 +CPU= 21.901
  21:28:01 STEP 1020 H=1020:00 +CPU= 45.087
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001020
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001020
  21:28:23 STEP 1021 H=1021:00 +CPU= 22.456
  21:28:46 STEP 1022 H=1022:00 +CPU= 22.990
  21:29:29 STEP 1023 H=1023:00 +CPU= 42.810
  21:29:57 STEP 1024 H=1024:00 +CPU= 27.877
  21:30:25 STEP 1025 H=1025:00 +CPU= 27.779
  21:31:09 STEP 1026 H=1026:00 +CPU= 44.163
  21:31:36 STEP 1027 H=1027:00 +CPU= 26.194
  21:32:04 STEP 1028 H=1028:00 +CPU= 27.892
  21:32:47 STEP 1029 H=1029:00 +CPU= 43.370
  21:33:13 STEP 1030 H=1030:00 +CPU= 25.624
  21:33:39 STEP 1031 H=1031:00 +CPU= 26.248
  21:34:28 STEP 1032 H=1032:00 +CPU= 49.165
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001032
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001008
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001020
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001020
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_42.zip
Uploading the intermediate file: upload_file_42.zip
  21:34:53 STEP 1033 H=1033:00 +CPU= 24.944
  21:35:17 STEP 1034 H=1034:00 +CPU= 23.212
  21:35:52 STEP 1035 H=1035:00 +CPU= 35.565
  21:36:15 STEP 1036 H=1036:00 +CPU= 22.524
  21:36:37 STEP 1037 H=1037:00 +CPU= 22.295
  21:37:13 STEP 1038 H=1038:00 +CPU= 36.292
  21:37:35 STEP 1039 H=1039:00 +CPU= 21.605
  21:37:58 STEP 1040 H=1040:00 +CPU= 23.545
  21:38:33 STEP 1041 H=1041:00 +CPU= 34.600
  21:38:55 STEP 1042 H=1042:00 +CPU= 22.191
  21:39:17 STEP 1043 H=1043:00 +CPU= 21.500
  21:39:58 STEP 1044 H=1044:00 +CPU= 41.298
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001044
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001044
  21:40:20 STEP 1045 H=1045:00 +CPU= 21.570
  21:40:42 STEP 1046 H=1046:00 +CPU= 22.116
  21:41:17 STEP 1047 H=1047:00 +CPU= 35.399
  21:41:39 STEP 1048 H=1048:00 +CPU= 22.149
  21:42:00 STEP 1049 H=1049:00 +CPU= 21.069
  21:42:35 STEP 1050 H=1050:00 +CPU= 34.745
  21:42:57 STEP 1051 H=1051:00 +CPU= 21.315
  21:43:19 STEP 1052 H=1052:00 +CPU= 22.083
  21:43:54 STEP 1053 H=1053:00 +CPU= 35.069
  21:44:16 STEP 1054 H=1054:00 +CPU= 22.503
  21:44:39 STEP 1055 H=1055:00 +CPU= 22.641
  21:45:24 STEP 1056 H=1056:00 +CPU= 45.095
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001056
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001032
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001044
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001044
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_43.zip
Uploading the intermediate file: upload_file_43.zip
  21:45:49 STEP 1057 H=1057:00 +CPU= 24.374
  21:46:13 STEP 1058 H=1058:00 +CPU= 24.094
  21:46:49 STEP 1059 H=1059:00 +CPU= 36.556
  21:47:14 STEP 1060 H=1060:00 +CPU= 24.168
  21:47:36 STEP 1061 H=1061:00 +CPU= 22.719
  21:48:14 STEP 1062 H=1062:00 +CPU= 37.809
  21:48:39 STEP 1063 H=1063:00 +CPU= 24.355
  21:49:03 STEP 1064 H=1064:00 +CPU= 24.733
  21:49:39 STEP 1065 H=1065:00 +CPU= 35.347
  21:50:02 STEP 1066 H=1066:00 +CPU= 23.456
  21:50:25 STEP 1067 H=1067:00 +CPU= 22.615
  21:51:05 STEP 1068 H=1068:00 +CPU= 39.796
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001068
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001068
  21:51:27 STEP 1069 H=1069:00 +CPU= 22.431
  21:51:51 STEP 1070 H=1070:00 +CPU= 23.489
  21:52:28 STEP 1071 H=1071:00 +CPU= 37.133
  21:52:52 STEP 1072 H=1072:00 +CPU= 24.251
  21:53:17 STEP 1073 H=1073:00 +CPU= 24.540
  21:53:53 STEP 1074 H=1074:00 +CPU= 36.192
  21:54:16 STEP 1075 H=1075:00 +CPU= 22.755
  21:54:40 STEP 1076 H=1076:00 +CPU= 24.317
  21:55:17 STEP 1077 H=1077:00 +CPU= 37.641
  21:55:41 STEP 1078 H=1078:00 +CPU= 23.580
  21:56:05 STEP 1079 H=1079:00 +CPU= 24.277
  21:56:50 STEP 1080 H=1080:00 +CPU= 44.227
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001080
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001056
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001068
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001068
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_44.zip
Uploading the intermediate file: upload_file_44.zip
  21:57:13 STEP 1081 H=1081:00 +CPU= 23.042
  21:57:37 STEP 1082 H=1082:00 +CPU= 24.654
  21:58:15 STEP 1083 H=1083:00 +CPU= 37.587
  21:58:39 STEP 1084 H=1084:00 +CPU= 23.665
  21:59:03 STEP 1085 H=1085:00 +CPU= 24.537
  21:59:40 STEP 1086 H=1086:00 +CPU= 36.655
  22:00:03 STEP 1087 H=1087:00 +CPU= 22.589
  22:00:28 STEP 1088 H=1088:00 +CPU= 25.702
  22:01:06 STEP 1089 H=1089:00 +CPU= 38.142
  22:01:33 STEP 1090 H=1090:00 +CPU= 26.251
  22:01:56 STEP 1091 H=1091:00 +CPU= 23.614
  22:02:39 STEP 1092 H=1092:00 +CPU= 42.988
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001092
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001092
  22:03:04 STEP 1093 H=1093:00 +CPU= 24.050
  22:03:29 STEP 1094 H=1094:00 +CPU= 25.037
  22:04:06 STEP 1095 H=1095:00 +CPU= 37.105
  22:04:32 STEP 1096 H=1096:00 +CPU= 26.157
  22:04:56 STEP 1097 H=1097:00 +CPU= 23.926
  22:05:35 STEP 1098 H=1098:00 +CPU= 39.685
  22:05:59 STEP 1099 H=1099:00 +CPU= 23.343
  22:06:23 STEP 1100 H=1100:00 +CPU= 23.947
  22:06:59 STEP 1101 H=1101:00 +CPU= 36.157
  22:07:24 STEP 1102 H=1102:00 +CPU= 24.962
  22:07:48 STEP 1103 H=1103:00 +CPU= 24.115
  22:08:33 STEP 1104 H=1104:00 +CPU= 44.587
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001104
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001080
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001092
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001092
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_45.zip
Uploading the intermediate file: upload_file_45.zip
  22:08:56 STEP 1105 H=1105:00 +CPU= 22.919
  22:09:21 STEP 1106 H=1106:00 +CPU= 25.156
  22:09:59 STEP 1107 H=1107:00 +CPU= 37.772
  22:10:23 STEP 1108 H=1108:00 +CPU= 24.581
  22:10:47 STEP 1109 H=1109:00 +CPU= 23.985
  22:11:26 STEP 1110 H=1110:00 +CPU= 38.514
  22:11:50 STEP 1111 H=1111:00 +CPU= 24.250
  22:12:15 STEP 1112 H=1112:00 +CPU= 25.153
  22:12:55 STEP 1113 H=1113:00 +CPU= 39.507
  22:13:20 STEP 1114 H=1114:00 +CPU= 25.109
  22:13:44 STEP 1115 H=1115:00 +CPU= 24.378
  22:14:26 STEP 1116 H=1116:00 +CPU= 42.001
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001116
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001116
  22:14:50 STEP 1117 H=1117:00 +CPU= 24.265
  22:15:15 STEP 1118 H=1118:00 +CPU= 24.584
  22:15:53 STEP 1119 H=1119:00 +CPU= 38.087
  22:16:18 STEP 1120 H=1120:00 +CPU= 25.048
  22:16:44 STEP 1121 H=1121:00 +CPU= 25.373
  22:17:22 STEP 1122 H=1122:00 +CPU= 38.387
  22:17:47 STEP 1123 H=1123:00 +CPU= 24.876
  22:18:12 STEP 1124 H=1124:00 +CPU= 25.503
  22:18:54 STEP 1125 H=1125:00 +CPU= 41.177
  22:19:20 STEP 1126 H=1126:00 +CPU= 26.632
  22:19:45 STEP 1127 H=1127:00 +CPU= 25.135
  22:20:32 STEP 1128 H=1128:00 +CPU= 46.306
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001128
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001104
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001116
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001116
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_46.zip
Uploading the intermediate file: upload_file_46.zip
  22:20:58 STEP 1129 H=1129:00 +CPU= 26.564
  22:21:24 STEP 1130 H=1130:00 +CPU= 25.560
  22:22:03 STEP 1131 H=1131:00 +CPU= 39.160
  22:22:28 STEP 1132 H=1132:00 +CPU= 25.232
  22:22:53 STEP 1133 H=1133:00 +CPU= 24.184
  22:23:32 STEP 1134 H=1134:00 +CPU= 39.662
  22:23:57 STEP 1135 H=1135:00 +CPU= 24.919
  22:24:23 STEP 1136 H=1136:00 +CPU= 25.494
  22:25:02 STEP 1137 H=1137:00 +CPU= 38.816
  22:25:27 STEP 1138 H=1138:00 +CPU= 25.837
  22:25:52 STEP 1139 H=1139:00 +CPU= 24.973
  22:26:39 STEP 1140 H=1140:00 +CPU= 46.892
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001140
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001140
  22:27:04 STEP 1141 H=1141:00 +CPU= 24.642
  22:27:28 STEP 1142 H=1142:00 +CPU= 23.729
  22:28:05 STEP 1143 H=1143:00 +CPU= 36.880
  22:28:29 STEP 1144 H=1144:00 +CPU= 24.117
  22:28:53 STEP 1145 H=1145:00 +CPU= 24.204
  22:29:33 STEP 1146 H=1146:00 +CPU= 39.204
  22:29:57 STEP 1147 H=1147:00 +CPU= 24.584
  22:30:21 STEP 1148 H=1148:00 +CPU= 24.163
  22:30:59 STEP 1149 H=1149:00 +CPU= 37.771
  22:31:23 STEP 1150 H=1150:00 +CPU= 23.812
  22:31:47 STEP 1151 H=1151:00 +CPU= 23.624
  22:32:32 STEP 1152 H=1152:00 +CPU= 44.923
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001152
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001128
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001140
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001140
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_47.zip
Uploading the intermediate file: upload_file_47.zip
  22:32:58 STEP 1153 H=1153:00 +CPU= 25.945
  22:33:24 STEP 1154 H=1154:00 +CPU= 25.834
  22:34:07 STEP 1155 H=1155:00 +CPU= 43.693
  22:34:32 STEP 1156 H=1156:00 +CPU= 24.283
  22:34:56 STEP 1157 H=1157:00 +CPU= 24.118
  22:35:40 STEP 1158 H=1158:00 +CPU= 43.653
  22:36:04 STEP 1159 H=1159:00 +CPU= 24.185
  22:36:28 STEP 1160 H=1160:00 +CPU= 24.168
  22:37:12 STEP 1161 H=1161:00 +CPU= 44.272
  22:37:39 STEP 1162 H=1162:00 +CPU= 26.742
  22:38:14 STEP 1163 H=1163:00 +CPU= 35.096
  22:39:06 STEP 1164 H=1164:00 +CPU= 51.308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001164
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001164
  22:39:31 STEP 1165 H=1165:00 +CPU= 25.356
  22:39:57 STEP 1166 H=1166:00 +CPU= 26.171
  22:40:37 STEP 1167 H=1167:00 +CPU= 39.931
  22:41:04 STEP 1168 H=1168:00 +CPU= 27.001
  22:41:30 STEP 1169 H=1169:00 +CPU= 25.880
  22:42:15 STEP 1170 H=1170:00 +CPU= 44.366
  22:42:41 STEP 1171 H=1171:00 +CPU= 26.015
  22:43:06 STEP 1172 H=1172:00 +CPU= 25.160
  22:43:48 STEP 1173 H=1173:00 +CPU= 42.614
  22:44:19 STEP 1174 H=1174:00 +CPU= 30.768
  22:44:49 STEP 1175 H=1175:00 +CPU= 29.250
  22:45:40 STEP 1176 H=1176:00 +CPU= 51.255
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001176
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001152
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001164
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001164
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_48.zip
Uploading the intermediate file: upload_file_48.zip
  22:46:05 STEP 1177 H=1177:00 +CPU= 25.586
  22:46:32 STEP 1178 H=1178:00 +CPU= 26.920
  22:47:12 STEP 1179 H=1179:00 +CPU= 39.944
  22:47:38 STEP 1180 H=1180:00 +CPU= 25.724
  22:48:05 STEP 1181 H=1181:00 +CPU= 27.226
  22:48:49 STEP 1182 H=1182:00 +CPU= 43.316
  22:49:15 STEP 1183 H=1183:00 +CPU= 26.190
  22:49:42 STEP 1184 H=1184:00 +CPU= 27.037
  22:50:23 STEP 1185 H=1185:00 +CPU= 41.277
  22:50:51 STEP 1186 H=1186:00 +CPU= 27.822
  22:51:19 STEP 1187 H=1187:00 +CPU= 28.168
  22:52:07 STEP 1188 H=1188:00 +CPU= 47.984
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001188
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001188
  22:52:34 STEP 1189 H=1189:00 +CPU= 26.534
  22:53:01 STEP 1190 H=1190:00 +CPU= 27.413
  22:53:44 STEP 1191 H=1191:00 +CPU= 42.730
  22:54:15 STEP 1192 H=1192:00 +CPU= 30.525
  22:54:41 STEP 1193 H=1193:00 +CPU= 26.487
  22:55:23 STEP 1194 H=1194:00 +CPU= 41.997
  22:55:49 STEP 1195 H=1195:00 +CPU= 25.720
  22:56:18 STEP 1196 H=1196:00 +CPU= 29.305
  22:57:02 STEP 1197 H=1197:00 +CPU= 44.216
  22:57:31 STEP 1198 H=1198:00 +CPU= 28.253
  22:57:58 STEP 1199 H=1199:00 +CPU= 27.082
  22:58:47 STEP 1200 H=1200:00 +CPU= 49.017
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001200
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001176
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001188
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001188
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_49.zip
Uploading the intermediate file: upload_file_49.zip
Uploading trickle at timestep: 4320000
  22:59:13 STEP 1201 H=1201:00 +CPU= 25.960
  22:59:42 STEP 1202 H=1202:00 +CPU= 28.704
  23:00:27 STEP 1203 H=1203:00 +CPU= 45.215
  23:00:55 STEP 1204 H=1204:00 +CPU= 27.974
  23:01:23 STEP 1205 H=1205:00 +CPU= 27.943
  23:02:05 STEP 1206 H=1206:00 +CPU= 42.534
  23:02:31 STEP 1207 H=1207:00 +CPU= 25.677
  23:02:58 STEP 1208 H=1208:00 +CPU= 27.252
  23:03:40 STEP 1209 H=1209:00 +CPU= 41.832
  23:04:05 STEP 1210 H=1210:00 +CPU= 24.999
  23:04:33 STEP 1211 H=1211:00 +CPU= 27.675
  23:05:18 STEP 1212 H=1212:00 +CPU= 45.516
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001212
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001212
  23:05:44 STEP 1213 H=1213:00 +CPU= 25.919
  23:06:11 STEP 1214 H=1214:00 +CPU= 26.648
  23:06:49 STEP 1215 H=1215:00 +CPU= 38.386
  23:07:14 STEP 1216 H=1216:00 +CPU= 25.002
  23:07:41 STEP 1217 H=1217:00 +CPU= 26.639
  23:08:23 STEP 1218 H=1218:00 +CPU= 41.729
  23:08:47 STEP 1219 H=1219:00 +CPU= 24.780
  23:09:13 STEP 1220 H=1220:00 +CPU= 25.483
  23:09:51 STEP 1221 H=1221:00 +CPU= 38.420
  23:10:16 STEP 1222 H=1222:00 +CPU= 25.043
  23:10:41 STEP 1223 H=1223:00 +CPU= 24.934
  23:11:26 STEP 1224 H=1224:00 +CPU= 44.440
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001224
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001200
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001212
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001212
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_50.zip
Uploading the intermediate file: upload_file_50.zip
  23:11:50 STEP 1225 H=1225:00 +CPU= 24.090
  23:12:14 STEP 1226 H=1226:00 +CPU= 23.976
  23:12:51 STEP 1227 H=1227:00 +CPU= 37.193
  23:13:19 STEP 1228 H=1228:00 +CPU= 27.670
  23:13:46 STEP 1229 H=1229:00 +CPU= 27.006
  23:14:32 STEP 1230 H=1230:00 +CPU= 45.836
  23:14:59 STEP 1231 H=1231:00 +CPU= 26.423
  23:15:26 STEP 1232 H=1232:00 +CPU= 27.603
  23:16:07 STEP 1233 H=1233:00 +CPU= 40.409
  23:16:34 STEP 1234 H=1234:00 +CPU= 27.120
  23:17:00 STEP 1235 H=1235:00 +CPU= 26.191
  23:17:45 STEP 1236 H=1236:00 +CPU= 44.781
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001236
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001236
  23:18:09 STEP 1237 H=1237:00 +CPU= 24.057
  23:18:33 STEP 1238 H=1238:00 +CPU= 24.409
  23:19:13 STEP 1239 H=1239:00 +CPU= 39.187
  23:19:39 STEP 1240 H=1240:00 +CPU= 26.822
  23:20:04 STEP 1241 H=1241:00 +CPU= 24.182
  23:20:46 STEP 1242 H=1242:00 +CPU= 42.616
  23:21:12 STEP 1243 H=1243:00 +CPU= 25.565
  23:21:38 STEP 1244 H=1244:00 +CPU= 25.698
  23:22:16 STEP 1245 H=1245:00 +CPU= 38.227
  23:22:42 STEP 1246 H=1246:00 +CPU= 26.524
  23:23:07 STEP 1247 H=1247:00 +CPU= 24.740
  23:23:52 STEP 1248 H=1248:00 +CPU= 45.038
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001248
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001224
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001236
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001236
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_51.zip
Uploading the intermediate file: upload_file_51.zip
  23:24:20 STEP 1249 H=1249:00 +CPU= 28.002
  23:24:45 STEP 1250 H=1250:00 +CPU= 24.836
  23:25:23 STEP 1251 H=1251:00 +CPU= 37.644
  23:25:48 STEP 1252 H=1252:00 +CPU= 25.592
  23:26:13 STEP 1253 H=1253:00 +CPU= 25.071
  23:26:52 STEP 1254 H=1254:00 +CPU= 38.627
  23:27:17 STEP 1255 H=1255:00 +CPU= 24.483
  23:27:42 STEP 1256 H=1256:00 +CPU= 25.336
  23:28:20 STEP 1257 H=1257:00 +CPU= 37.664
  23:28:43 STEP 1258 H=1258:00 +CPU= 23.727
  23:29:07 STEP 1259 H=1259:00 +CPU= 23.248
  23:29:50 STEP 1260 H=1260:00 +CPU= 43.706
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001260
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001260
  23:30:17 STEP 1261 H=1261:00 +CPU= 26.167
  23:30:45 STEP 1262 H=1262:00 +CPU= 28.256
  23:31:28 STEP 1263 H=1263:00 +CPU= 42.966
  23:31:57 STEP 1264 H=1264:00 +CPU= 29.159
  23:32:23 STEP 1265 H=1265:00 +CPU= 26.306
  23:33:05 STEP 1266 H=1266:00 +CPU= 41.318
  23:33:32 STEP 1267 H=1267:00 +CPU= 27.577
  23:34:01 STEP 1268 H=1268:00 +CPU= 28.252
  23:34:46 STEP 1269 H=1269:00 +CPU= 45.119
  23:35:16 STEP 1270 H=1270:00 +CPU= 30.498
  23:35:46 STEP 1271 H=1271:00 +CPU= 29.500
  23:36:36 STEP 1272 H=1272:00 +CPU= 50.264
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001272
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001248
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001260
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001260
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_52.zip
Uploading the intermediate file: upload_file_52.zip
  23:37:03 STEP 1273 H=1273:00 +CPU= 26.553
  23:37:32 STEP 1274 H=1274:00 +CPU= 28.807
  23:38:14 STEP 1275 H=1275:00 +CPU= 41.964
  23:38:40 STEP 1276 H=1276:00 +CPU= 26.649
  23:39:06 STEP 1277 H=1277:00 +CPU= 25.305
  23:39:48 STEP 1278 H=1278:00 +CPU= 42.668
  23:40:15 STEP 1279 H=1279:00 +CPU= 26.750
  23:40:43 STEP 1280 H=1280:00 +CPU= 28.020
  23:41:28 STEP 1281 H=1281:00 +CPU= 44.725
  23:41:56 STEP 1282 H=1282:00 +CPU= 28.086
  23:42:22 STEP 1283 H=1283:00 +CPU= 25.705
  23:43:08 STEP 1284 H=1284:00 +CPU= 46.462
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001284
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001284
  23:43:34 STEP 1285 H=1285:00 +CPU= 25.315
  23:44:00 STEP 1286 H=1286:00 +CPU= 26.146
  23:44:39 STEP 1287 H=1287:00 +CPU= 39.255
  23:45:05 STEP 1288 H=1288:00 +CPU= 26.120
  23:45:30 STEP 1289 H=1289:00 +CPU= 25.209
  23:46:12 STEP 1290 H=1290:00 +CPU= 41.427
  23:46:40 STEP 1291 H=1291:00 +CPU= 27.864
  23:47:12 STEP 1292 H=1292:00 +CPU= 31.937
  23:48:01 STEP 1293 H=1293:00 +CPU= 49.134
  23:48:37 STEP 1294 H=1294:00 +CPU= 36.400
  23:49:11 STEP 1295 H=1295:00 +CPU= 33.839
  23:50:07 STEP 1296 H=1296:00 +CPU= 55.594
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001296
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001272
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001284
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001284
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_53.zip
Uploading the intermediate file: upload_file_53.zip
  23:50:37 STEP 1297 H=1297:00 +CPU= 30.507
  23:51:11 STEP 1298 H=1298:00 +CPU= 33.725
  23:51:55 STEP 1299 H=1299:00 +CPU= 44.020
  23:52:26 STEP 1300 H=1300:00 +CPU= 30.449
  23:52:55 STEP 1301 H=1301:00 +CPU= 29.967
  23:53:44 STEP 1302 H=1302:00 +CPU= 48.408
  23:54:15 STEP 1303 H=1303:00 +CPU= 30.871
  23:54:45 STEP 1304 H=1304:00 +CPU= 30.030
  23:55:31 STEP 1305 H=1305:00 +CPU= 45.995
  23:56:01 STEP 1306 H=1306:00 +CPU= 30.538
  23:56:31 STEP 1307 H=1307:00 +CPU= 29.483
  23:57:21 STEP 1308 H=1308:00 +CPU= 50.069
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001308
  23:57:50 STEP 1309 H=1309:00 +CPU= 28.595
  23:58:21 STEP 1310 H=1310:00 +CPU= 31.025
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0900
(argv4) batchid: 982
(argv5) wuid: 12199544
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c3634359645249baf4258e112dca9f87 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0900_2013050100_123_982_12199544.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0900_2013050100_123_982_12199544.zip
ic_ancil_file: ic_ancil_12199544
ifsdata_file: ifsdata_12199544
climate_data_file: clim_data_12199544
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_12737ecbadf171fabb13a6de814bbfc7 to: /var/lib/boinc-client/slots/4/ic_ancil_12199544.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12199544.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b333a64a7caabbe26130998c4d806ec7 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12199544.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12199544.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12199544.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12199544.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12199544.xml
Opened progress file ok : /var/lib/boinc-client/slots/4/progress_file_12199544.xml
-- Model is restarting --
Adjusting last_iter, 1311, to previous model restart step.
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12199544.xml
last_cpu_time: 42957
upload_file_number: 54
last_iter: 1295
last_upload: 4665600
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397
The child process has been launched with process id: 349661
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:349661:349661] [20230215:060006:1676412006:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
Quit request received from BOINC client, ending the child process
(argv0) ../../projects/climateprediction.net/oifs_43r3_ps_1.05_x86_64-pc-linux-gnu
(argv1) start_date: 2013050100
(argv2) exptid: hq0f
(argv3) unique_member_id: 0900
(argv4) batchid: 982
(argv5) wuid: 12199544
(argv6) fclen: 123
(argv7) app_name: oifs_43r3_ps
(argv8) nthreads: 1
Working directory is: /var/lib/boinc-client/slots/4
Project directory is: /var/lib/boinc-client/projects/climateprediction.net/
app name: oifs_43r3_ps
version: 1.05
Location of temp folder: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544
..mkdir for temp folder for results failed
Copying: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Unzipping the app zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_app_1.05_x86_64-pc-linux-gnu.zip
Copying the namelist files from: ../../projects/climateprediction.net/jf_c3634359645249baf4258e112dca9f87 to: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0900_2013050100_123_982_12199544.zip
Unzipping the namelist zip file: /var/lib/boinc-client/slots/4/oifs_43r3_ps_0900_2013050100_123_982_12199544.zip
ic_ancil_file: ic_ancil_12199544
ifsdata_file: ifsdata_12199544
climate_data_file: clim_data_12199544
horiz_resolution: 159
vert_resolution: 60
grid_type: l_2
upload_interval: 24
utstep: 3600
nfrres: restart dump frequency (steps) 24
Copying IC ancils from: ../../projects/climateprediction.net/jf_12737ecbadf171fabb13a6de814bbfc7 to: /var/lib/boinc-client/slots/4/ic_ancil_12199544.zip
Unzipping the IC ancils zip file: /var/lib/boinc-client/slots/4/ic_ancil_12199544.zip
..mkdir for ifsdata folder failed
Copying the ifsdata_file from: ../../projects/climateprediction.net/jf_b333a64a7caabbe26130998c4d806ec7 to: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12199544.zip
Unzipping the ifsdata_zip file: /var/lib/boinc-client/slots/4/ifsdata/ifsdata_12199544.zip
..mkdir for the climate data folder failed
Copying the climate data file from: ../../projects/climateprediction.net/jf_f32c668cf7829426ffd94699b48b9a2e to: /var/lib/boinc-client/slots/4/159l_2/clim_data_12199544.zip
Unzipping the climate data zip file: /var/lib/boinc-client/slots/4/159l_2/clim_data_12199544.zip
Checking for progress XML file: /var/lib/boinc-client/slots/4/progress_file_12199544.xml
Creating progress file: /var/lib/boinc-client/slots/4/progress_file_12199544.xml
last_cpu_time: 0
upload_file_number: 0
last_iter: 0
last_upload: 0
model_completed: 0
total_length_of_simulation: 10627200
result_base_name: oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397
The child process has been launched with process id: 354663
Executing the command: /var/lib/boinc-client/slots/4/oifs_43r3_model.exe
[EC_DRHOOK:hostname:myproc:omptid:pid:unixtid] [YYYYMMDD:HHMMSS:epoch:walltime] [function@file:lineno] -- Max OpenMP threads = 1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2091] fp = 0x3129fe0
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2098] DR_HOOK_ALLOW_COREDUMP=-1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2104] Hardlimit for core file is now 0 (0x0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2122] DR_HOOK_PROFILE_PROC=-1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2128] DR_HOOK_PROFILE_LIMIT=-10.000
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2194] DR_HOOK_RANDOM_MEMSTAT=0  (RAND_MAX=2147483647)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2205] DR_HOOK_HASHBITS=16
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2213] DR_HOOK_NCALLSTACK=0
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2221] DR_HOOK_HARAKIRI_TIMEOUT=500
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2228] DR_HOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2235] DR_HOOK_TRAPFPE_INVALID=1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2242] DR_HOOK_TRAPFPE_DIVBYZERO=1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [process_options@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:2249] DR_HOOK_TRAPFPE_OVERFLOW=1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [ignore_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1227] DR_HOOK_IGNORE_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [restore_default_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1178] DR_HOOK_RESTORE_DEFAULT_SIGNALS=<undef>
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1937] New signal handler 'signal_drhook' for signal#6 (SIGABRT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1938] New signal handler 'signal_drhook' for signal#7 (SIGBUS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1939] New signal handler 'signal_drhook' for signal#11 (SIGSEGV) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1944] New signal handler 'signal_drhook' for signal#16 (SIGSTKFLT) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [trapfpe_treatment@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1149] DR_HOOK enables SIGFPE-related floating point trapping since DRHOOK_TRAPFPE=1
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1948] New signal handler 'signal_drhook' for signal#8 (SIGFPE) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1949] New signal handler 'signal_drhook' for signal#4 (SIGILL) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1951] New signal handler 'signal_drhook' for signal#5 (SIGTRAP) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1952] New signal handler 'signal_drhook' for signal#2 (SIGINT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1960] New signal handler 'signal_drhook' for signal#3 (SIGQUIT) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1961] New signal handler 'signal_drhook' for signal#15 (SIGTERM) at 0x81f820 (old at 0x1ce8bb0)
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1966] New signal handler 'signal_drhook' for signal#24 (SIGXCPU) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1968] New signal handler 'signal_drhook' for signal#25 (SIGXFSZ) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [signal_drhook_init@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1973] New signal handler 'signal_drhook' for signal#31 (SIGSYS) at 0x81f820 (old at (nil))
[EC_DRHOOK:taman-jurong:1:1:354663:354663] [20230215:071759:1676416679:0.000] [catch_signals@/home/glenn/github/jamie_oifs43r3.git/src/ifsaux/support/drhook.c:1111] DR_HOOK_CATCH_SIGNALS=<undef>
  07:18:40 STEP 1296 H=1296:00 +CPU= 38.139
  07:19:02 STEP 1297 H=1297:00 +CPU= 21.362
  07:19:23 STEP 1298 H=1298:00 +CPU= 21.281
  07:19:57 STEP 1299 H=1299:00 +CPU= 33.819
  07:20:18 STEP 1300 H=1300:00 +CPU= 21.139
  07:20:38 STEP 1301 H=1301:00 +CPU= 20.150
  07:21:11 STEP 1302 H=1302:00 +CPU= 32.713
  07:21:31 STEP 1303 H=1303:00 +CPU= 20.514
  07:21:53 STEP 1304 H=1304:00 +CPU= 21.835
  07:22:29 STEP 1305 H=1305:00 +CPU= 35.718
  07:22:54 STEP 1306 H=1306:00 +CPU= 25.159
  07:23:18 STEP 1307 H=1307:00 +CPU= 23.870
  07:24:00 STEP 1308 H=1308:00 +CPU= 41.547
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001308
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001308
  07:24:25 STEP 1309 H=1309:00 +CPU= 24.895
  07:24:48 STEP 1310 H=1310:00 +CPU= 23.439
  07:25:24 STEP 1311 H=1311:00 +CPU= 35.474
  07:25:48 STEP 1312 H=1312:00 +CPU= 24.205
  07:26:12 STEP 1313 H=1313:00 +CPU= 24.548
  07:26:51 STEP 1314 H=1314:00 +CPU= 38.130
  07:27:15 STEP 1315 H=1315:00 +CPU= 24.471
  07:27:39 STEP 1316 H=1316:00 +CPU= 24.281
  07:28:18 STEP 1317 H=1317:00 +CPU= 37.991
  07:28:43 STEP 1318 H=1318:00 +CPU= 25.023
  07:29:07 STEP 1319 H=1319:00 +CPU= 24.523
  07:29:52 STEP 1320 H=1320:00 +CPU= 44.727
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001320
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001296
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001308
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001308
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_1.zip
Uploading the intermediate file: upload_file_1.zip
  07:30:16 STEP 1321 H=1321:00 +CPU= 23.962
  07:30:40 STEP 1322 H=1322:00 +CPU= 24.137
  07:31:18 STEP 1323 H=1323:00 +CPU= 38.360
  07:31:43 STEP 1324 H=1324:00 +CPU= 24.948
  07:32:08 STEP 1325 H=1325:00 +CPU= 24.062
  07:32:46 STEP 1326 H=1326:00 +CPU= 37.993
  07:33:09 STEP 1327 H=1327:00 +CPU= 23.452
  07:33:32 STEP 1328 H=1328:00 +CPU= 22.932
  07:34:07 STEP 1329 H=1329:00 +CPU= 35.028
  07:34:30 STEP 1330 H=1330:00 +CPU= 23.381
  07:34:54 STEP 1331 H=1331:00 +CPU= 23.237
  07:35:34 STEP 1332 H=1332:00 +CPU= 40.403
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001332
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001332
  07:35:56 STEP 1333 H=1333:00 +CPU= 22.251
  07:36:20 STEP 1334 H=1334:00 +CPU= 23.067
  07:36:55 STEP 1335 H=1335:00 +CPU= 35.624
  07:37:20 STEP 1336 H=1336:00 +CPU= 24.451
  07:37:43 STEP 1337 H=1337:00 +CPU= 23.520
  07:38:21 STEP 1338 H=1338:00 +CPU= 38.049
  07:38:44 STEP 1339 H=1339:00 +CPU= 22.602
  07:39:07 STEP 1340 H=1340:00 +CPU= 23.214
  07:39:44 STEP 1341 H=1341:00 +CPU= 37.082
  07:40:09 STEP 1342 H=1342:00 +CPU= 24.660
  07:40:32 STEP 1343 H=1343:00 +CPU= 22.827
  07:41:13 STEP 1344 H=1344:00 +CPU= 41.315
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001344
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001320
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001332
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001332
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_2.zip
Uploading the intermediate file: upload_file_2.zip
  07:41:35 STEP 1345 H=1345:00 +CPU= 21.994
  07:41:57 STEP 1346 H=1346:00 +CPU= 22.093
  07:42:34 STEP 1347 H=1347:00 +CPU= 36.461
  07:42:58 STEP 1348 H=1348:00 +CPU= 24.105
  07:43:22 STEP 1349 H=1349:00 +CPU= 23.580
  07:44:00 STEP 1350 H=1350:00 +CPU= 38.207
  07:44:23 STEP 1351 H=1351:00 +CPU= 22.881
  07:44:46 STEP 1352 H=1352:00 +CPU= 23.631
  07:45:24 STEP 1353 H=1353:00 +CPU= 37.623
  07:45:48 STEP 1354 H=1354:00 +CPU= 23.732
  07:46:11 STEP 1355 H=1355:00 +CPU= 22.885
  07:46:49 STEP 1356 H=1356:00 +CPU= 38.743
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001356
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001356
  07:47:12 STEP 1357 H=1357:00 +CPU= 22.623
  07:47:37 STEP 1358 H=1358:00 +CPU= 24.516
  07:48:14 STEP 1359 H=1359:00 +CPU= 37.725
  07:48:38 STEP 1360 H=1360:00 +CPU= 23.531
  07:49:01 STEP 1361 H=1361:00 +CPU= 22.850
  07:49:38 STEP 1362 H=1362:00 +CPU= 36.868
  07:50:00 STEP 1363 H=1363:00 +CPU= 22.509
  07:50:23 STEP 1364 H=1364:00 +CPU= 22.822
  07:51:00 STEP 1365 H=1365:00 +CPU= 36.515
  07:51:24 STEP 1366 H=1366:00 +CPU= 23.933
  07:51:47 STEP 1367 H=1367:00 +CPU= 23.752
  07:52:31 STEP 1368 H=1368:00 +CPU= 43.569
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMGGhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMSHhq0f+001368
Moving to projects directory: /var/lib/boinc-client/slots/4/ICMUAhq0f+001368
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001344
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMGGhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMSHhq0f+001356
Adding to the zip: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_12199544/ICMUAhq0f+001356
Zipping up the intermediate file: /var/lib/boinc-client/projects/climateprediction.net/oifs_43r3_ps_0900_2013050100_123_982_12199544_0_r1094654397_3.zip
Uploading the intermediate file: upload_file_3.zip
  07:52:54 STEP 1369 H=1369:00 +CPU= 23.135
Abort request received from BOINC client, ending the child process

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 May 2023 16:51:28 1534673 22285731 oifs_43r3_ps_0900_2013050100_123_982_12199544_0 4,320,000 39,350 0.0091
09 May 2023 16:51:00 1534673 22285731 oifs_43r3_ps_0900_2013050100_123_982_12199544_0 2,505,600 22,988 0.0092


©2024 cpdn.org