Task 12831763

Name	hadcm3n_p6fi_1900_40_007225918_0
Workunit	7424158
Created	26 Apr 2011, 15:36:06 UTC
Sent	27 Apr 2011, 11:45:11 UTC
Report deadline	27 Jul 2011, 19:12:22 UTC
Received	30 May 2011, 21:26:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125747
Run time	13 days 4 hours 8 min 48 sec
CPU time	6 days 9 hours 5 min 41 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11532, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8620, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5496, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:01:06 (2960): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 May 2011 15:10:55	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	596,160	990,580	1.6616
29 May 2011 19:49:15	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	570,240	950,931	1.6676
28 May 2011 15:51:39	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	544,320	911,368	1.6743
27 May 2011 19:44:49	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	518,400	871,823	1.6818
26 May 2011 18:47:56	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	492,480	832,042	1.6895
26 May 2011 00:35:45	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	466,560	792,531	1.6987
24 May 2011 17:09:28	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	440,640	750,387	1.7029
24 May 2011 04:16:57	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	414,720	709,876	1.7117
20 May 2011 06:15:41	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	388,800	669,614	1.7223
19 May 2011 07:32:49	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	362,880	627,896	1.7303
17 May 2011 15:14:10	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	336,960	587,667	1.7440
16 May 2011 00:28:02	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	311,040	535,800	1.7226
06 May 2011 10:06:03	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	285,120	495,134	1.7366
04 May 2011 02:29:27	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	259,200	440,265	1.6986
03 May 2011 09:34:50	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	233,280	395,404	1.6950
02 May 2011 20:11:33	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	207,360	353,846	1.7064
01 May 2011 20:02:03	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	181,440	311,620	1.7175
01 May 2011 07:19:26	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	155,520	269,824	1.7350
30 Apr 2011 17:14:57	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	129,600	227,919	1.7586
30 Apr 2011 04:15:25	1125747	12831763	hadcm3n_p6fi_1900_40_007225918_0	103,680	187,048	1.8041