climateprediction.net home page
Task 12735992

Task 12735992

Name hadcm3n_o1ll_1900_40_007197404_1
Workunit 7395684
Created 28 Mar 2011, 14:00:46 UTC
Sent 1 Apr 2011, 16:01:20 UTC
Report deadline 1 Jul 2011, 23:28:31 UTC
Received 27 Apr 2011, 12:32:39 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1116505
Run time 16 days 9 hours 52 min 38 sec
CPU time 10 days 19 hours 52 min 48 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 2.46 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
06:55:04 (1011): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
15:12:59 (17272): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:54:02 (30202): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:54:03 (30202): No heartbeat from core client for 30 sec - exiting
Suspended CPDN Monitor - Suspend request from BOINC...
12:57:22 (52779): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:58:42 (37755): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
12:51:44 (43995): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
19:16:55 (408): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
19:16:57 (408): No heartbeat from core client for 30 sec - exiting
01:25:29 (8967): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
00:05:02 (20856): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
04:07:13 (64626): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
03:50:01 (72575): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
10:24:30 (18500): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:26:49 (31137): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
11:26:50 (31137): No heartbeat from core client for 30 sec - exiting
16:48:17 (25803): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
17:08:21 (9432): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:29:42 (48278): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
08:29:43 (48278): No heartbeat from core client for 30 sec - exiting
08:29:44 (48278): No heartbeat from core client for 30 sec - exiting
08:29:45 (48278): No heartbeat from core client for 30 sec - exiting
08:29:46 (48278): No heartbeat from core client for 30 sec - exiting
08:29:47 (48278): No heartbeat from core client for 30 sec - exiting
08:29:48 (48278): No heartbeat from core client for 30 sec - exiting
08:29:49 (48278): No heartbeat from core client for 30 sec - exiting
08:29:50 (48278): No heartbeat from core client for 30 sec - exiting
08:29:51 (48278): No heartbeat from core client for 30 sec - exiting
08:29:52 (48278): No heartbeat from core client for 30 sec - exiting
08:29:53 (48278): No heartbeat from core client for 30 sec - exiting
08:29:54 (48278): No heartbeat from core client for 30 sec - exiting
08:29:55 (48278): No heartbeat from core client for 30 sec - exiting
08:29:56 (48278): No heartbeat from core client for 30 sec - exiting
08:29:57 (48278): No heartbeat from core client for 30 sec - exiting
08:29:58 (48278): No heartbeat from core client for 30 sec - exiting
08:29:59 (48278): No heartbeat from core client for 30 sec - exiting
08:30:00 (48278): No heartbeat from core client for 30 sec - exiting
08:30:01 (48278): No heartbeat from core client for 30 sec - exiting
08:30:02 (48278): No heartbeat from core client for 30 sec - exiting
08:30:03 (48278): No heartbeat from core client for 30 sec - exiting
08:30:04 (48278): No heartbeat from core client for 30 sec - exiting
08:30:05 (48278): No heartbeat from core client for 30 sec - exiting
08:30:06 (48278): No heartbeat from core client for 30 sec - exiting
08:30:07 (48278): No heartbeat from core client for 30 sec - exiting
08:30:08 (48278): No heartbeat from core client for 30 sec - exiting
08:30:09 (48278): No heartbeat from core client for 30 sec - exiting
08:30:10 (48278): No heartbeat from core client for 30 sec - exiting
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:03 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:12 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:19 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:23 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:26 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
SIGABRT: abort called

Crashed executable name: hadcm3n_um_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.7 build 10J869
Wed Apr 27 08:31:29 2011

Thread 0 Crashed:
0   libSystem.B.dylib                   0x9194e156 __kill + 10
1   libSystem.B.dylib                   0x919e0899 raise + 26
2   libSystem.B.dylib                   0x919f69b8 abort + 93
3   hadcm3n_um_6.07_i686-apple-darwin   0x00336cff um_setup + 127
4   hadcm3n_um_6.07_i686-apple-darwin   0x00389b71 um_shell + 3953
5   hadcm3n_um_6.07_i686-apple-darwin   0x000048e1 main + 1153
6   hadcm3n_um_6.07_i686-apple-darwin   0x000027fb _start + 209
7   hadcm3n_um_6.07_i686-apple-darwin   0x00002729 start + 41

Thread 1:
0   libSystem.B.dylib                   0x918ed1ae mach_wait_until + 10
1   libSystem.B.dylib                   0x91974809 nanosleep + 345
2   libSystem.B.dylib                   0x919746aa usleep + 61
3   hadcm3n_um_6.07_i686-apple-darwin   0x00442ec0 boinc_sleep(double) + 188
4   hadcm3n_um_6.07_i686-apple-darwin   0x0043865e timer_thread(void*) + 78
5   libSystem.B.dylib                   0x9191a7fd _pthread_start + 345
6   libSystem.B.dylib                   0x9191a682 thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffca28 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x00447dc2  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -   0x4cbfff /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_um_6.07_i686-apple-darwin
0x918ec000 - 0x91a93fff /usr/lib/libSystem.B.dylib
0x9797f000 - 0x97982fff /usr/lib/system/libmathCommon.A.dylib
0x98338000 - 0x98346fff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=86378, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
26 Apr 2011 21:26:53 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 518,400 898,558 1.7333
26 Apr 2011 04:29:20 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 492,480 855,603 1.7373
25 Apr 2011 10:56:22 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 466,560 812,808 1.7421
24 Apr 2011 17:38:58 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 440,640 769,999 1.7475
24 Apr 2011 01:23:00 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 414,720 727,671 1.7546
23 Apr 2011 09:03:16 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 388,800 685,185 1.7623
22 Apr 2011 16:11:04 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 362,880 642,331 1.7701
21 Apr 2011 21:45:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 336,960 598,808 1.7771
21 Apr 2011 03:04:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 311,040 555,086 1.7846
20 Apr 2011 18:21:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 285,120 511,590 1.7943
20 Apr 2011 18:21:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 259,200 467,260 1.8027
20 Apr 2011 18:21:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 233,280 423,500 1.8154
20 Apr 2011 18:21:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 207,360 379,863 1.8319
20 Apr 2011 18:21:19 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 181,440 336,538 1.8548
20 Apr 2011 18:21:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 155,520 291,937 1.8772
20 Apr 2011 18:21:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 129,600 246,927 1.9053
20 Apr 2011 18:21:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 103,680 200,798 1.9367
20 Apr 2011 18:21:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 77,760 152,403 1.9599
12 Apr 2011 21:38:18 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 51,840 103,229 1.9913
11 Apr 2011 23:39:03 1116505 12735992 hadcm3n_o1ll_1900_40_007197404_1 25,920 52,297 2.0176


©2024 climateprediction.net