Task 11554570

Name	famous_ulj0_1399_200_006660743_2
Workunit	6864115
Created	10 Jun 2010, 15:04:18 UTC
Sent	10 Jul 2010, 23:11:23 UTC
Report deadline	10 Oct 2010, 6:38:34 UTC
Received	7 Aug 2010, 12:40:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1086753
Run time	5 days 5 hours 46 min 40 sec
CPU time	4 days 19 hours 34 min 45 sec
Validate state	Invalid
Credit	2,995.60
Device peak FLOPS	2.68 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.11.2</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Signal 1 received, exiting... (29911): called boinc_finish Signal 1 received, exiting... (29906): called boinc_finish Signal 15 received, exiting... (29906): called boinc_finish Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2340): called boinc_finish Signal 15 received, exiting... (2340): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (8941): called boinc_finish Signal 15 received, exiting... (8941): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (11454): called boinc_finish Signal 15 received, exiting... (11454): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2269): called boinc_finish Signal 1 received, exiting... (2253): called boinc_finish Signal 15 received, exiting... (2253): called boinc_finish Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Signal 1 received, exiting... (2480): called boinc_finish Signal 15 received, exiting... (2480): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2340): called boinc_finish Signal 15 received, exiting... (2340): called boinc_finish BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2339): called boinc_finish Signal 15 received, exiting... (2339): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2273): called boinc_finish Signal 15 received, exiting... (2273): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2282): called boinc_finish Signal 15 received, exiting... (2282): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2543): called boinc_finish Signal 15 received, exiting... (2543): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2340): called boinc_finish Signal 1 received, exiting... (2324): called boinc_finish Signal 15 received, exiting... (2324): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2355): called boinc_finish Signal 1 received, exiting... (2338): called boinc_finish Signal 15 received, exiting... (2338): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2351): called boinc_finish Signal 15 received, exiting... (2351): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Signal 1 received, exiting... (2682): called boinc_finish Signal 15 received, exiting... (2682): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 1 received, exiting... Signal 1 received, exiting... (22562): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (22807): called boinc_finish Signal 15 received, exiting... (22807): called boinc_finish BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2260): called boinc_finish Signal 15 received, exiting... (2260): called boinc_finish BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Signal 1 received, exiting... (2305): called boinc_finish Signal 15 received, exiting... (2305): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Signal 1 received, exiting... (3049): called boinc_finish Signal 15 received, exiting... (3049): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2384): called boinc_finish Signal 1 received, exiting... (2362): called boinc_finish Signal 15 received, exiting... (2362): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Signal 15 received, exiting... (2296): called boinc_finish (2296): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2321): called boinc_finish Signal 15 received, exiting... (2321): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... (16947): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2463): called boinc_finish Signal 1 received, exiting... (2433): called boinc_finish Signal 15 received, exiting... (2433): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2272): called boinc_finish Signal 1 received, exiting... (2226): called boinc_finish Signal 15 received, exiting... (2226): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2290): called boinc_finish Signal 15 received, exiting... (2290): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2278): called boinc_finish Signal 15 received, exiting... (2278): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (4148): called boinc_finish Signal 15 received, exiting... (4148): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (10137): called boinc_finish Signal 15 received, exiting... (10137): called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... (2341): called boinc_finish Signal 15 received, exiting... (2341): called boinc_finish SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 SETPOS: Seek Failed: Invalid argument SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 61 to Word Address -198 Failed with Error Code -1 Sorry, too many model crashes! :-( (2321): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Aug 2010 12:20:21	1086753	11554570	famous_ulj0_1399_200_006660743_2	907,946	412,695	0.4545
06 Aug 2010 10:00:08	1086753	11554570	famous_ulj0_1399_200_006660743_2	898,586	408,448	0.4545
05 Aug 2010 20:17:57	1086753	11554570	famous_ulj0_1399_200_006660743_2	889,226	404,199	0.4546
05 Aug 2010 19:03:23	1086753	11554570	famous_ulj0_1399_200_006660743_2	879,866	399,950	0.4546
05 Aug 2010 17:26:21	1086753	11554570	famous_ulj0_1399_200_006660743_2	870,506	395,707	0.4546
05 Aug 2010 16:05:09	1086753	11554570	famous_ulj0_1399_200_006660743_2	861,146	391,455	0.4546
05 Aug 2010 14:54:53	1086753	11554570	famous_ulj0_1399_200_006660743_2	851,786	387,205	0.4546
05 Aug 2010 12:34:47	1086753	11554570	famous_ulj0_1399_200_006660743_2	842,426	382,960	0.4546
05 Aug 2010 11:19:52	1086753	11554570	famous_ulj0_1399_200_006660743_2	833,066	378,713	0.4546
05 Aug 2010 08:46:11	1086753	11554570	famous_ulj0_1399_200_006660743_2	823,706	374,469	0.4546
04 Aug 2010 21:09:41	1086753	11554570	famous_ulj0_1399_200_006660743_2	814,346	370,211	0.4546
04 Aug 2010 19:59:01	1086753	11554570	famous_ulj0_1399_200_006660743_2	804,986	365,977	0.4546
04 Aug 2010 19:48:01	1086753	11554570	famous_ulj0_1399_200_006660743_2	795,626	361,740	0.4547
04 Aug 2010 17:39:17	1086753	11554570	famous_ulj0_1399_200_006660743_2	786,266	357,508	0.4547
04 Aug 2010 16:27:12	1086753	11554570	famous_ulj0_1399_200_006660743_2	776,906	353,275	0.4547
04 Aug 2010 15:20:40	1086753	11554570	famous_ulj0_1399_200_006660743_2	767,546	349,037	0.4547
04 Aug 2010 14:05:02	1086753	11554570	famous_ulj0_1399_200_006660743_2	758,186	344,795	0.4548
04 Aug 2010 12:47:46	1086753	11554570	famous_ulj0_1399_200_006660743_2	748,826	340,560	0.4548
03 Aug 2010 18:58:47	1086753	11554570	famous_ulj0_1399_200_006660743_2	739,466	336,316	0.4548
03 Aug 2010 04:47:33	1086753	11554570	famous_ulj0_1399_200_006660743_2	730,106	332,068	0.4548