Task 15790767

Name	hadcm3n_zkul_1920_40_008366186_1
Workunit	8517045
Created	20 May 2013, 13:01:47 UTC
Sent	20 May 2013, 13:02:05 UTC
Report deadline	19 Aug 2013, 20:29:16 UTC
Received	14 Jun 2013, 14:35:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153653
Run time	24 days 6 hours 56 min 17 sec
CPU time	15 days 9 hours 42 min 56 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	1.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 12:45:42 (22293): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:49:44 (24169): No heartbeat from core client for 30 sec - exiting 20:49:45 (24169): No heartbeat from core client for 30 sec - exiting 20:49:46 (24169): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/zkulko.pjc5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulko.pjc5c10 Error: Input file: dataout/zkulko.pic5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulko.pic5c10 Error: Input file: dataout/zkulko.pfc5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulko.pfc5c10 Error: Input file: dataout/zkulka.phc5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulka.phc5c10 Error: Input file: dataout/zkulka.pgc5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulka.pgc5c10 Error: Input file: dataout/zkulka.pec5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulka.pec5c10 Error: Input file: dataout/zkulka.pdc5c10 is not a valid UM file. Error converting file to netcdf: dataout/zkulka.pdc5c10 CPDN Monitor - Quit request from BOINC... 01:29:48 (25563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:05 (26699): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:28 (26731): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:38:15 (26763): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:42:26 (26795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:50 (26827): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:44 (26859): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7797400] [0xb7797430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7621651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb7624a82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb760dbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb774c400] [0xb774c430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb75d6651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb75d9a82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb75c2bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7881400] [0xb7881430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb770b651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb770ea82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76f7bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7829400] [0xb7829430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb76b3651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb76b6a82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb769fbd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb7888400] [0xb7888430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7712651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb7715a82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb76febd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xb77bd400] [0xb77bd430] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xb7647651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xb764aa82] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7633bd6] /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=30112, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jun 2013 01:03:39	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	388,800	1,288,600	3.3143
13 Jun 2013 02:06:17	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	362,880	1,213,228	3.3433
12 Jun 2013 02:03:02	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	336,960	1,134,380	3.3665
10 Jun 2013 15:20:39	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	311,040	1,049,111	3.3729
09 Jun 2013 18:08:46	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	285,120	977,267	3.4276
08 Jun 2013 20:49:09	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	259,200	905,469	3.4933
07 Jun 2013 13:57:41	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	233,280	823,159	3.5286
05 Jun 2013 22:56:36	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	207,360	733,785	3.5387
04 Jun 2013 01:20:31	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	181,440	640,227	3.5286
01 Jun 2013 22:29:47	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	155,520	544,502	3.5012
31 May 2013 03:04:00	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	129,600	455,128	3.5118
29 May 2013 09:33:11	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	103,680	364,729	3.5178
27 May 2013 04:12:18	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	77,760	273,298	3.5146
25 May 2013 09:52:28	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	51,840	184,102	3.5514
23 May 2013 02:56:01	1153653	15790767	hadcm3n_zkul_1920_40_008366186_1	25,920	91,010	3.5112