Task 15698440

Name	hadcm3n_3ikf_1940_40_008262768_2
Workunit	8417892
Created	1 Apr 2013, 8:46:18 UTC
Sent	1 Apr 2013, 8:46:27 UTC
Report deadline	1 Jul 2013, 16:13:38 UTC
Received	17 Apr 2013, 20:32:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1253297
Run time	9 days 21 hours 45 min 39 sec
CPU time	6 days 8 hours 31 min 17 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.89 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... OPEN: File Creation Failed: Too many open files in system OPEN: Unable to Open File dataout/3ikfko.dag09i0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 SIGBUS: bus error built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) Mon Apr 15 19:16:57 2013 Thread 0 Crashed: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc538 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x904f0000 - 0x904f7fff /usr/lib/system/libsystem_dnssd.dylib 0x90544000 - 0x90549fff /usr/lib/system/libmacho.dylib 0x912e2000 - 0x912e3fff /usr/lib/system/libremovefile.dylib 0x913ed000 - 0x913f1fff /usr/lib/system/libcache.dylib 0x914a9000 - 0x91574fff /usr/lib/system/libsystem_c.dylib 0x91a7b000 - 0x91a7cfff /usr/lib/system/libsystem_sandbox.dylib 0x923c8000 - 0x923d6fff /usr/lib/libz.1.dylib 0x923ed000 - 0x923fbfff /usr/lib/system/libdispatch.dylib 0x9306d000 - 0x9309cfff /usr/lib/system/libsystem_info.dylib 0x93100000 - 0x93100fff /usr/lib/system/libkeymgr.dylib 0x9310c000 - 0x9310ffff /usr/lib/system/libmathCommon.A.dylib 0x931d5000 - 0x931ddfff /usr/lib/system/liblaunch.dylib 0x9346e000 - 0x93472fff /usr/lib/system/libsystem_network.dylib 0x95831000 - 0x95839fff /usr/lib/system/libcopyfile.dylib 0x95b9a000 - 0x95bddfff /usr/lib/system/libcommonCrypto.dylib 0x95e6f000 - 0x95e77fff /usr/lib/system/libunwind.dylib 0x95eda000 - 0x95ee1fff /usr/lib/system/libsystem_notify.dylib 0x97bbb000 - 0x97bd1fff /usr/lib/system/libxpc.dylib 0x9813c000 - 0x9813dfff /usr/lib/system/libquarantine.dylib 0x98277000 - 0x98278fff /usr/lib/system/libunc.dylib 0x985aa000 - 0x985abfff /usr/lib/system/libsystem_blocks.dylib 0x985b4000 - 0x985b6fff /usr/lib/system/libdyld.dylib 0x99754000 - 0x99782fff /usr/lib/libSystem.B.dylib 0x99838000 - 0x9983bfff /usr/lib/system/libcompiler_rt.dylib 0x99d01000 - 0x99d1ffff /usr/lib/system/libsystem_kernel.dylib 0x9a05b000 - 0x9a05bfff /usr/lib/system/libdnsinfo.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=39725, iMonCtr=1 Model crash detected, will try to restart... Model crashed: jobs/climate.cpdc SIGABRT: abort called built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) Mon Apr 15 19:17:01 2013 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffccd8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2 cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin 0x904f0000 - 0x904f7fff /usr/lib/system/libsystem_dnssd.dylib 0x90544000 - 0x90549fff /usr/lib/system/libmacho.dylib 0x912e2000 - 0x912e3fff /usr/lib/system/libremovefile.dylib 0x913ed000 - 0x913f1fff /usr/lib/system/libcache.dylib 0x914a9000 - 0x91574fff /usr/lib/system/libsystem_c.dylib 0x91a7b000 - 0x91a7cfff /usr/lib/system/libsystem_sandbox.dylib 0x923c8000 - 0x923d6fff /usr/lib/libz.1.dylib 0x923ed000 - 0x923fbfff /usr/lib/system/libdispatch.dylib 0x9306d000 - 0x9309cfff /usr/lib/system/libsystem_info.dylib 0x93100000 - 0x93100fff /usr/lib/system/libkeymgr.dylib 0x9310c000 - 0x9310ffff /usr/lib/system/libmathCommon.A.dylib 0x931d5000 - 0x931ddfff /usr/lib/system/liblaunch.dylib 0x9346e000 - 0x93472fff /usr/lib/system/libsystem_network.dylib 0x95831000 - 0x95839fff /usr/lib/system/libcopyfile.dylib 0x95b9a000 - 0x95bddfff /usr/lib/system/libcommonCrypto.dylib 0x95e6f000 - 0x95e77fff /usr/lib/system/libunwind.dylib 0x95eda000 - 0x95ee1fff /usr/lib/system/libsystem_notify.dylib 0x97bbb000 - 0x97bd1fff /usr/lib/system/libxpc.dylib 0x9813c000 - 0x9813dfff /usr/lib/system/libquarantine.dylib 0x98277000 - 0x98278fff /usr/lib/system/libunc.dylib 0x985aa000 - 0x985abfff /usr/lib/system/libsystem_blocks.dylib 0x985b4000 - 0x985b6fff /usr/lib/system/libdyld.dylib 0x99754000 - 0x99782fff /usr/lib/libSystem.B.dylib 0x99838000 - 0x9983bfff /usr/lib/system/libcompiler_rt.dylib 0x99d01000 - 0x99d1ffff /usr/lib/system/libsystem_kernel.dylib 0x9a05b000 - 0x9a05bfff /usr/lib/system/libdnsinfo.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=39725, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... OPEN: File Creation Failed: Too many open files in system OPEN: Unable to Open File dataout/3ikfka.dag09m0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=39725, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_se_6.07_i686-apple-darwin.dylib after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3ikf_1940_40_008262768/jobs/xabnk.ihist after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3ikf_1940_40_008262768/jobs/xabnk.namelists after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3ikf_1940_40_008262768/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_3ikf_1940_40_008262768/dataout/ocean_restart.day after 11 attempts SIGBUS: bus error built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) Mon Apr 15 12:27:44 2013 Thread 0 Crashed: Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc0f8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x904f0000 - 0x904f7fff /usr/lib/system/libsystem_dnssd.dylib 0x90544000 - 0x90549fff /usr/lib/system/libmacho.dylib 0x9054a000 - 0x905acfff /usr/lib/libstdc++.6.dylib 0x912e2000 - 0x912e3fff /usr/lib/system/libremovefile.dylib 0x913ed000 - 0x913f1fff /usr/lib/system/libcache.dylib 0x914a9000 - 0x91574fff /usr/lib/system/libsystem_c.dylib 0x91a7b000 - 0x91a7cfff /usr/lib/system/libsystem_sandbox.dylib 0x923c8000 - 0x923d6fff /usr/lib/libz.1.dylib 0x923ed000 - 0x923fbfff /usr/lib/system/libdispatch.dylib 0x9306d000 - 0x9309cfff /usr/lib/system/libsystem_info.dylib 0x93100000 - 0x93100fff /usr/lib/system/libkeymgr.dylib 0x9310c000 - 0x9310ffff /usr/lib/system/libmathCommon.A.dylib 0x931d5000 - 0x931ddfff /usr/lib/system/liblaunch.dylib 0x9346e000 - 0x93472fff /usr/lib/system/libsystem_network.dylib 0x95831000 - 0x95839fff /usr/lib/system/libcopyfile.dylib 0x95b9a000 - 0x95bddfff /usr/lib/system/libcommonCrypto.dylib 0x95e6f000 - 0x95e77fff /usr/lib/system/libunwind.dylib 0x95eda000 - 0x95ee1fff /usr/lib/system/libsystem_notify.dylib 0x97711000 - 0x9771afff /usr/lib/libc++abi.dylib 0x97bbb000 - 0x97bd1fff /usr/lib/system/libxpc.dylib 0x9813c000 - 0x9813dfff /usr/lib/system/libquarantine.dylib 0x98277000 - 0x98278fff /usr/lib/system/libunc.dylib 0x985aa000 - 0x985abfff /usr/lib/system/libsystem_blocks.dylib 0x985b4000 - 0x985b6fff /usr/lib/system/libdyld.dylib 0x99754000 - 0x99782fff /usr/lib/libSystem.B.dylib 0x99838000 - 0x9983bfff /usr/lib/system/libcompiler_rt.dylib 0x99d01000 - 0x99d1ffff /usr/lib/system/libsystem_kernel.dylib 0x9a05b000 - 0x9a05bfff /usr/lib/system/libdnsinfo.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Apr 2013 20:33:05	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	492,480	748,789	1.5204
17 Apr 2013 20:33:05	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	466,560	708,470	1.5185
17 Apr 2013 20:33:04	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	440,640	668,125	1.5163
17 Apr 2013 20:33:04	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	414,720	627,815	1.5138
13 Apr 2013 06:12:41	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	388,800	587,463	1.5110
12 Apr 2013 17:02:19	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	362,880	546,996	1.5074
12 Apr 2013 06:03:25	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	336,960	511,604	1.5183
10 Apr 2013 12:29:31	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	311,040	476,911	1.5333
09 Apr 2013 22:08:59	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	285,120	442,270	1.5512
09 Apr 2013 09:10:07	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	259,200	405,004	1.5625
08 Apr 2013 09:27:22	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	233,280	364,259	1.5615
07 Apr 2013 21:22:37	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	207,360	323,924	1.5621
07 Apr 2013 08:32:49	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	181,440	283,431	1.5621
06 Apr 2013 16:26:11	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	155,520	242,674	1.5604
05 Apr 2013 23:03:37	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	129,600	202,142	1.5597
05 Apr 2013 10:20:25	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	103,680	161,525	1.5579
04 Apr 2013 09:05:46	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	77,760	121,285	1.5597
03 Apr 2013 11:21:35	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	51,840	80,820	1.5590
02 Apr 2013 09:59:03	1253297	15698440	hadcm3n_3ikf_1940_40_008262768_2	25,920	40,484	1.5619