Task 16573720

Name	hadam3pm2_e5j0_1991_10_008710207_1
Workunit	8856681
Created	19 Apr 2014, 9:55:29 UTC
Sent	19 Apr 2014, 13:23:44 UTC
Report deadline	1 Apr 2015, 18:43:44 UTC
Received	12 Jul 2014, 17:54:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1325086
Run time	19 days 14 hours 58 min 5 sec
CPU time	10 days 19 hours 8 min 6 sec
Validate state	Invalid
Credit	8,900.08
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office HadAM3P (global only) with MOSES II landsurface scheme v7.03 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|2dec.nc oa.pc\|3jan.nc oa.pc\|3feb.nc oa.pc\|3mar.nc oa.pc\|3apr.nc oa.pc\|3may.nc oa.pc\|3jun.nc oa.pc\|3jul.nc oa.pc\|3aug.nc oa.pc\|3sep.nc oa.pc\|3oct.nc oa.pc\|3nov.nc oa.pe\|2dec.nc oa.pe\|3jan.nc oa.pe\|3feb.nc oa.pe\|3mar.nc oa.pe\|3apr.nc oa.pe\|3may.nc oa.pe\|3jun.nc oa.pe\|3jul.nc oa.pe\|3aug.nc oa.pe\|3sep.nc oa.pe\|3oct.nc oa.pe\|3nov.nc oa.pc\|3dec.nc oa.pc\|4jan.nc oa.pc\|4feb.nc oa.pc\|4mar.nc oa.pc\|4apr.nc oa.pc\|4may.nc oa.pc\|4jun.nc oa.pc\|4jul.nc oa.pc\|4aug.nc oa.pc\|4sep.nc oa.pc\|4oct.nc oa.pc\|4nov.nc oa.pe\|3dec.nc oa.pe\|4jan.nc oa.pe\|4feb.nc oa.pe\|4mar.nc oa.pe\|4apr.nc oa.pe\|4may.nc oa.pe\|4jun.nc oa.pe\|4jul.nc oa.pe\|4aug.nc oa.pe\|4sep.nc oa.pe\|4oct.nc oa.pe\|4nov.nc Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:29:44 (3584): No heartbeat from client for 30 sec - exiting 23:29:44 (3584): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:35:02 (15621): No heartbeat from client for 30 sec - exiting 00:35:02 (15621): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:42:54 (15699): No heartbeat from client for 30 sec - exiting 00:42:54 (15699): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:01 (15733): No heartbeat from client for 30 sec - exiting 00:50:01 (15733): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:52:58 (15760): No heartbeat from client for 30 sec - exiting 00:52:58 (15760): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:56:19 (15787): No heartbeat from client for 30 sec - exiting 00:56:19 (15787): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:02 (15814): No heartbeat from client for 30 sec - exiting 00:59:02 (15814): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:37 (15840): No heartbeat from client for 30 sec - exiting 01:01:37 (15840): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:15:16 (15872): No heartbeat from client for 30 sec - exiting 01:15:16 (15872): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:17:56 (15903): No heartbeat from client for 30 sec - exiting 01:17:56 (15903): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:22 (15929): No heartbeat from client for 30 sec - exiting 01:20:22 (15929): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:22:37 (15953): No heartbeat from client for 30 sec - exiting 01:22:37 (15953): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:27:22 (15976): No heartbeat from client for 30 sec - exiting 01:27:22 (15976): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:30 (16000): No heartbeat from client for 30 sec - exiting 01:29:30 (16000): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:34:14 (16023): No heartbeat from client for 30 sec - exiting 01:34:14 (16023): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:35:52 (16048): No heartbeat from client for 30 sec - exiting 01:35:52 (16048): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:39:14 (16080): No heartbeat from client for 30 sec - exiting 01:39:14 (16080): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:45 (16101): No heartbeat from client for 30 sec - exiting 01:41:45 (16101): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:52:15 (16127): No heartbeat from client for 30 sec - exiting 01:52:15 (16127): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:20 (16153): No heartbeat from client for 30 sec - exiting 01:54:20 (16153): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:05:09 (16176): No heartbeat from client for 30 sec - exiting 02:05:09 (16176): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:00 (16205): No heartbeat from client for 30 sec - exiting 02:07:00 (16205): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:09:14 (16229): No heartbeat from client for 30 sec - exiting 02:09:14 (16229): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:53 (16252): No heartbeat from client for 30 sec - exiting 02:16:53 (16252): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:29:07 (16280): No heartbeat from client for 30 sec - exiting 02:29:07 (16280): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:33:29 (16306): No heartbeat from client for 30 sec - exiting 02:33:29 (16306): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:42:04 (16333): No heartbeat from client for 30 sec - exiting 02:42:04 (16333): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:50:08 (16364): No heartbeat from client for 30 sec - exiting 02:50:08 (16364): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:34 (16388): No heartbeat from client for 30 sec - exiting 02:57:34 (16388): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:00:09 (16413): No heartbeat from client for 30 sec - exiting 03:00:09 (16413): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:13 (16436): No heartbeat from client for 30 sec - exiting 03:02:13 (16436): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:54 (16463): No heartbeat from client for 30 sec - exiting 03:02:54 (16463): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:04 (16484): No heartbeat from client for 30 sec - exiting 03:05:04 (16484): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:13 (16508): No heartbeat from client for 30 sec - exiting 03:12:13 (16508): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:14:54 (16534): No heartbeat from client for 30 sec - exiting 03:14:54 (16534): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:17:30 (16558): No heartbeat from client for 30 sec - exiting 03:17:30 (16558): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:20:51 (16582): No heartbeat from client for 30 sec - exiting 03:20:51 (16582): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:24:57 (16608): No heartbeat from client for 30 sec - exiting 03:24:57 (16608): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:31 (16631): No heartbeat from client for 30 sec - exiting 03:27:31 (16631): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:34:06 (16655): No heartbeat from client for 30 sec - exiting 03:34:06 (16655): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:44 (16682): No heartbeat from client for 30 sec - exiting 03:36:44 (16682): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:56 (16718): No heartbeat from client for 30 sec - exiting 03:38:56 (16718): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:57 (16741): No heartbeat from client for 30 sec - exiting 03:41:57 (16741): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:30 (16765): No heartbeat from client for 30 sec - exiting 03:44:30 (16765): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:47:06 (16786): No heartbeat from client for 30 sec - exiting 03:47:06 (16786): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:35 (16812): No heartbeat from client for 30 sec - exiting 03:49:35 (16812): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:01:27 (16835): No heartbeat from client for 30 sec - exiting 04:01:27 (16835): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:49 (16865): No heartbeat from client for 30 sec - exiting 04:02:49 (16865): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:19 (16885): No heartbeat from client for 30 sec - exiting 04:05:19 (16885): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:44 (16906): No heartbeat from client for 30 sec - exiting 04:08:44 (16906): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:45 (16926): No heartbeat from client for 30 sec - exiting 04:09:45 (16926): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:22 (16944): No heartbeat from client for 30 sec - exiting 04:11:22 (16944): timer handler: client dead, exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|7dec.nc oa.pc\|8jan.nc oa.pc\|8feb.nc oa.pc\|8mar.nc oa.pc\|8apr.nc oa.pc\|8may.nc oa.pc\|8jun.nc oa.pc\|8jul.nc oa.pc\|8aug.nc oa.pc\|8sep.nc oa.pc\|8oct.nc oa.pc\|8nov.nc oa.pe\|7dec.nc oa.pe\|8jan.nc oa.pe\|8feb.nc oa.pe\|8mar.nc oa.pe\|8apr.nc oa.pe\|8may.nc oa.pe\|8jun.nc oa.pe\|8jul.nc oa.pe\|8aug.nc oa.pe\|8sep.nc oa.pe\|8oct.nc oa.pe\|8nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|8dec.nc oa.pc\|9jan.nc oa.pc\|9feb.nc oa.pc\|9mar.nc oa.pc\|9apr.nc oa.pc\|9may.nc oa.pc\|9jun.nc oa.pc\|9jul.nc oa.pc\|9aug.nc oa.pc\|9sep.nc oa.pc\|9oct.nc oa.pc\|9nov.nc oa.pe\|8dec.nc oa.pe\|9jan.nc oa.pe\|9feb.nc oa.pe\|9mar.nc oa.pe\|9apr.nc oa.pe\|9may.nc oa.pe\|9jun.nc oa.pe\|9jul.nc oa.pe\|9aug.nc oa.pe\|9sep.nc oa.pe\|9oct.nc oa.pe\|9nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... oa.pc\|9dec.nc oa.pc\|0jan.nc oa.pc\|0feb.nc oa.pc\|0mar.nc oa.pc\|0apr.nc oa.pc\|0may.nc oa.pc\|0jun.nc oa.pc\|0jul.nc oa.pc\|0aug.nc oa.pc\|0sep.nc oa.pc\|0oct.nc oa.pc\|0nov.nc oa.pe\|9dec.nc oa.pe\|0jan.nc oa.pe\|0feb.nc oa.pe\|0mar.nc oa.pe\|0apr.nc oa.pe\|0may.nc oa.pe\|0jun.nc oa.pe\|0jul.nc oa.pe\|0aug.nc oa.pe\|0sep.nc oa.pe\|0oct.nc oa.pe\|0nov.nc Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: æM Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Model crashed: æM Sorry, too many model crashes! :-( 19:51:33 (3549): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2014 16:24:02	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	308,228	1,645,280	5.3379
06 Jul 2014 02:46:01	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	305,348	1,624,097	5.3188
04 Jul 2014 17:20:11	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	302,468	1,603,332	5.3008
02 Jul 2014 18:48:28	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	299,588	1,582,816	5.2833
01 Jul 2014 22:43:58	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	296,708	1,563,504	5.2695
30 Jun 2014 03:24:32	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	293,828	1,544,430	5.2562
28 Jun 2014 02:33:05	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	290,948	1,526,309	5.2460
27 Jun 2014 04:50:49	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	288,068	1,506,681	5.2303
24 Jun 2014 16:43:38	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	285,188	1,487,999	5.2176
21 Jun 2014 15:01:31	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	282,308	1,469,769	5.2063
21 Jun 2014 01:49:27	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	279,428	1,459,259	5.2223
20 Jun 2014 22:23:55	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	273,668	1,447,958	5.2909
20 Jun 2014 19:23:25	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	270,788	1,437,225	5.3076
20 Jun 2014 05:00:46	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	267,908	1,417,473	5.2909
17 Jun 2014 04:50:23	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	265,028	1,397,991	5.2749
16 Jun 2014 10:23:31	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	262,148	1,378,170	5.2572
15 Jun 2014 08:26:13	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	259,268	1,358,124	5.2383
14 Jun 2014 15:04:46	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	256,388	1,338,252	5.2196
14 Jun 2014 06:08:51	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	253,508	1,317,767	5.1981
13 Jun 2014 08:21:49	1325086	16573720	hadam3pm2_e5j0_1991_10_008710207_1	250,628	1,297,138	5.1756