Task 12877908

Name	hadcm3n_p4nr_1900_40_007223623_2
Workunit	7421863
Created	9 May 2011, 21:42:33 UTC
Sent	9 May 2011, 21:43:31 UTC
Report deadline	9 Aug 2011, 5:10:42 UTC
Received	28 May 2011, 16:20:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1132206
Run time	9 days 12 hours 45 min 39 sec
CPU time	8 days 23 hours 34 min 35 sec
Validate state	Invalid
Credit	6,531.84
Device peak FLOPS	3.25 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x1851004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x1851000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x2831004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x284c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x284c600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x284c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x284c600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x1835a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x1851004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x1851000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x2028604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x833604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x833600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6964,0xa09b1540) malloc: * error for object 0x824e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/p4nrko.pjb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrko.pjb3c10 Error: Input file: dataout/p4nrko.pib3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrko.pib3c10 Error: Input file: dataout/p4nrko.pfb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrko.pfb3c10 Error: Input file: dataout/p4nrka.phb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrka.phb3c10 Error: Input file: dataout/p4nrka.pgb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrka.pgb3c10 Error: Input file: dataout/p4nrka.peb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrka.peb3c10 Error: Input file: dataout/p4nrka.pdb3c10 is not a valid UM file. Error converting file to netcdf: dataout/p4nrka.pdb3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... execl(/Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin, 135980) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10573, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 May 2011 07:24:35	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	544,320	745,714	1.3700
26 May 2011 17:45:19	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	518,400	710,217	1.3700
26 May 2011 05:59:55	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	492,480	674,959	1.3705
25 May 2011 17:13:42	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	466,560	639,488	1.3706
25 May 2011 03:45:11	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	440,640	604,526	1.3719
24 May 2011 15:11:13	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	414,720	569,097	1.3722
24 May 2011 04:52:08	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	388,800	534,417	1.3745
22 May 2011 23:43:40	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	362,880	498,401	1.3735
22 May 2011 05:31:30	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	336,960	461,657	1.3701
21 May 2011 03:32:10	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	311,040	426,039	1.3697
20 May 2011 04:52:25	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	285,120	390,196	1.3685
19 May 2011 17:14:09	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	259,200	355,056	1.3698
19 May 2011 06:11:30	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	233,280	319,832	1.3710
18 May 2011 18:55:30	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	207,360	285,132	1.3751
18 May 2011 04:10:03	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	181,440	249,636	1.3759
17 May 2011 17:13:39	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	155,520	214,835	1.3814
16 May 2011 23:15:09	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	129,600	179,798	1.3873
14 May 2011 22:35:42	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	103,680	144,981	1.3984
14 May 2011 01:34:35	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	77,760	110,416	1.4200
12 May 2011 17:14:13	1132206	12877908	hadcm3n_p4nr_1900_40_007223623_2	51,840	75,106	1.4488