Task 15869206

Name	hadcm3n_4dds_2020_40_008396598_2
Workunit	8547457
Created	27 Jun 2013, 8:29:11 UTC
Sent	27 Jun 2013, 8:40:49 UTC
Report deadline	26 Sep 2013, 16:08:00 UTC
Received	2 Nov 2013, 22:14:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1202292
Run time	6 days 17 hours 56 min 18 sec
CPU time	5 days 21 hours 22 min 6 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid 22:20:33 (80148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:20:34 (80148): No heartbeat from core client for 30 sec - exiting 22:20:35 (80148): No heartbeat from core client for 30 sec - exiting 22:20:36 (80148): No heartbeat from core client for 30 sec - exiting 22:20:37 (80148): No heartbeat from core client for 30 sec - exiting 22:20:38 (80148): No heartbeat from core client for 30 sec - exiting 22:20:39 (80148): No heartbeat from core client for 30 sec - exiting 22:20:40 (80148): No heartbeat from core client for 30 sec - exiting 22:20:41 (80148): No heartbeat from core client for 30 sec - exiting dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid 12:28:28 (11994): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:28:29 (11994): No heartbeat from core client for 30 sec - exiting dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid 08:16:02 (91163): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid 09:13:42 (6343): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(30480,0xaca9ea28) malloc: * error for object 0x2b5ca00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/7/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/4ddsko.pjn3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddsko.pjn3c10 Error: Input file: dataout/4ddsko.pin3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddsko.pin3c10 Error: Input file: dataout/4ddsko.pfn3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddsko.pfn3c10 Error: Input file: dataout/4ddska.phn3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddska.phn3c10 Error: Input file: dataout/4ddska.pgn3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddska.pgn3c10 Error: Input file: dataout/4ddska.pen3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddska.pen3c10 Error: Input file: dataout/4ddska.pdn3c10 is not a valid UM file. Error converting file to netcdf: dataout/4ddska.pdn3c10 CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 22:08:44 (26522): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: Result too large BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: Result too large BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: Result too large BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: Result too large BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 BUFFOUT: Write Failed: Result too large BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Nov 2013 11:19:08	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	336,960	482,235	1.4311
28 Oct 2013 23:43:45	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	311,040	446,094	1.4342
27 Oct 2013 21:54:07	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	285,120	411,311	1.4426
23 Oct 2013 14:03:40	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	259,200	373,028	1.4392
23 Oct 2013 14:03:40	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	233,280	337,635	1.4473
02 Sep 2013 12:15:37	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	207,360	299,283	1.4433
22 Aug 2013 00:11:01	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	181,440	265,367	1.4626
21 Aug 2013 08:26:41	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	155,520	232,503	1.4950
20 Aug 2013 19:36:25	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	129,600	197,127	1.5210
14 Aug 2013 21:05:32	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	103,680	160,223	1.5454
14 Aug 2013 21:05:32	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	77,760	121,692	1.5650
14 Aug 2013 21:05:32	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	51,840	80,263	1.5483
14 Aug 2013 21:05:32	1202292	15869206	hadcm3n_4dds_2020_40_008396598_2	25,920	36,465	1.4068