Task 12986719

Name	hadcm3n_s03c_1940_40_007298184_2
Workunit	7495608
Created	20 Jun 2011, 12:09:35 UTC
Sent	20 Jun 2011, 12:09:45 UTC
Report deadline	19 Sep 2011, 19:36:56 UTC
Received	4 Jul 2011, 8:39:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	431496
Run time	9 days 23 hours 37 min 59 sec
CPU time	7 days 15 hours 20 min 4 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	1.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:49:36 (6252): No heartbeat from core client for 30 sec - exiting 16:49:38 (6252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:39 (6252): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:16:28 (3748): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:07:23 (7512): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s03cko.pje5c10 Error converting file to netcdf: dataout/s03cko.pie5c10 Error converting file to netcdf: dataout/s03cko.pfe5c10 Error converting file to netcdf: dataout/s03cka.phe5c10 Error converting file to netcdf: dataout/s03cka.pge5c10 Error converting file to netcdf: dataout/s03cka.pee5c10 Error converting file to netcdf: dataout/s03cka.pde5c10 Suspended CPDN Monitor - Suspend request from BOINC... 18:39:25 (6932): No heartbeat from core client for 30 sec - exiting 18:39:26 (6932): No heartbeat from core client for 30 sec - exiting 18:39:27 (6932): No heartbeat from core client for 30 sec - exiting 18:39:28 (6932): No heartbeat from core client for 30 sec - exiting 18:39:29 (6932): No heartbeat from core client for 30 sec - exiting 18:39:30 (6932): No heartbeat from core client for 30 sec - exiting 18:39:31 (6932): No heartbeat from core client for 30 sec - exiting 18:39:32 (6932): No heartbeat from core client for 30 sec - exiting 18:39:34 (6932): No heartbeat from core client for 30 sec - exiting 18:39:35 (6932): No heartbeat from core client for 30 sec - exiting 18:39:36 (6932): No heartbeat from core client for 30 sec - exiting 18:39:37 (6932): No heartbeat from core client for 30 sec - exiting 18:39:38 (6932): No heartbeat from core client for 30 sec - exiting 18:39:39 (6932): No heartbeat from core client for 30 sec - exiting 18:39:40 (6932): No heartbeat from core client for 30 sec - exiting 18:39:41 (6932): No heartbeat from core client for 30 sec - exiting 18:39:42 (6932): No heartbeat from core client for 30 sec - exiting 18:39:43 (6932): No heartbeat from core client for 30 sec - exiting 18:39:44 (6932): No heartbeat from core client for 30 sec - exiting 18:39:46 (6932): No heartbeat from core client for 30 sec - exiting 18:39:47 (6932): No heartbeat from core client for 30 sec - exiting 18:39:48 (6932): No heartbeat from core client for 30 sec - exiting 18:39:49 (6932): No heartbeat from core client for 30 sec - exiting 18:39:50 (6932): No heartbeat from core client for 30 sec - exiting 18:39:51 (6932): No heartbeat from core client for 30 sec - exiting 18:39:52 (6932): No heartbeat from core client for 30 sec - exiting 18:39:53 (6932): No heartbeat from core client for 30 sec - exiting 18:39:54 (6932): No heartbeat from core client for 30 sec - exiting 18:39:55 (6932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s03cko.pje5c10 Error converting file to netcdf: dataout/s03cko.pie5c10 Error converting file to netcdf: dataout/s03cko.pfe5c10 Error converting file to netcdf: dataout/s03cka.phe5c10 Error converting file to netcdf: dataout/s03cka.pge5c10 Error converting file to netcdf: dataout/s03cka.pee5c10 Error converting file to netcdf: dataout/s03cka.pde5c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:52:14 (3348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:52:15 (3348): No heartbeat from core client for 30 sec - exiting 09:38:35 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:58 (9316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Jul 2011 05:36:28	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	233,280	804,657	3.4493
03 Jul 2011 03:53:57	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	207,360	716,357	3.4547
02 Jul 2011 00:20:20	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	181,440	626,976	3.4556
30 Jun 2011 20:02:42	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	155,520	537,223	3.4544
29 Jun 2011 17:20:17	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	129,600	447,941	3.4563
28 Jun 2011 04:51:43	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	103,680	357,577	3.4489
27 Jun 2011 04:12:31	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	77,760	269,660	3.4678
22 Jun 2011 18:31:02	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	51,840	180,356	3.4791
21 Jun 2011 15:34:12	431496	12986719	hadcm3n_s03c_1940_40_007298184_2	25,920	89,951	3.4703