Christoph Eberhardt | 16 May 14:12 2012

Startup fails after updgrading from 1.0.8 to 1.1.0

Hi there,

if updgraded cassandra from 1.0.8 to 1.1.0. It seemed to work in the first place, all seemed to work fine. So I
started upgrading the rest of the cluster (at the time only one other node, that is a replica). After having
a several errors, I restarted the cluster and now cassandra won't even start up. Startup fails with the
following error message: 

INFO 13:59:05,175 Logging initialized INFO 13:59:05,178 JVM vendor/version: Java HotSpot(TM) 64-Bit
Server VM/1.6.0_26 INFO 13:59:05,178 Heap size: 8420720640/8420720640 INFO 13:59:05,178 Classpath:
bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.1.0.jar:bin/../lib/apache-cassandra-clientutil-1.1.0.jar:bin/../lib/apache-cassandra-thrift-1.1.0.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:bin/../lib/guava-r08.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/metrics-core-2.0.3.jar:bin/../lib/mx4j-tools.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.6.1.jar:bin/../lib/slf4j-log4j12-1.6.1.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.4.1.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar
INFO 13:59:07,312 JNA mlockall successful INFO 13:59:07,325 Loading settings from
file:/opt/cassandra/apache-cassandra-1.1.0/conf/cassandra.yaml INFO 13:59:07,419
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 13:59:07,567 Global
memtable threshold is enabled at 2676MB
 INFO 13:59:07,654 Initializing key cache with capacity of 100 MBs.
 INFO 13:59:07,661 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 13:59:07,662 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 13:59:07,664 Scheduling row cache save to each 0 seconds (going to save all keys).
 INFO 13:59:07,717 Opening
/opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-9
(28520 bytes)
 INFO 13:59:07,717 Opening
/opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-10
(28520 bytes)
 INFO 13:59:07,746 Opening
/opt/cassandra/database/data/system/NodeIdInfo/system-NodeIdInfo-hc-1 (187 bytes)
 INFO 13:59:07,777 Opening
/opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-3 (1892 bytes)
 INFO 13:59:07,777 Opening
/opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-1 (1892 bytes)
(Continue reading)

Dave Brosius | 16 May 15:14 2012

Re: Startup fails after updgrading from 1.0.8 to 1.1.0

Might be related to

https://issues.apache.org/jira/browse/CASSANDRA-3794



On 05/16/2012 08:12 AM, Christoph Eberhardt wrote:
Hi there, if updgraded cassandra from 1.0.8 to 1.1.0. It seemed to work in the first place, all seemed to work fine. So I started upgrading the rest of the cluster (at the time only one other node, that is a replica). After having a several errors, I restarted the cluster and now cassandra won't even start up. Startup fails with the following error message: INFO 13:59:05,175 Logging initialized INFO 13:59:05,178 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 13:59:05,178 Heap size: 8420720640/8420720640 INFO 13:59:05,178 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.1.0.jar:bin/../lib/apache-cassandra-clientutil-1.1.0.jar:bin/../lib/apache-cassandra-thrift-1.1.0.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:bin/../lib/guava-r08.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl- 1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/metrics-core-2 . 0.3.jar:bin/../lib/mx4j-tools.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.6.1.jar:bin/../lib/slf4j-log4j12-1.6.1.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.4.1.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar INFO 13:59:07,312 JNA mlockall successful INFO 13:59:07,325 Loading settings from file:/opt/cassandra/apache-cassandra-1.1.0/conf/cassandra.yaml INFO 13:59:07,419 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 13:59:07,567 Global memtable threshold is enabled at 2676MB INFO 13:59:07,654 Initializing key cache with capacity of 100 MBs. INFO 13:59:07,661 Scheduling key cache save to each 14400 seconds (going to save all keys). INFO 13:59:07,662 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO 13:59:07,664 Scheduling row cache save to each 0 seconds (going to save all keys). INFO 13:59:07,717 Opening /opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-9 (28520 bytes) INFO 13:59:07,717 Opening /opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-10 (28520 bytes) INFO 13:59:07,746 Opening /opt/cassandra/database/data/system/NodeIdInfo/system-NodeIdInfo-hc-1 (187 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-3 (1892 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-1 (1892 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-2 (1892 bytes) INFO 13:59:07,790 Opening /opt/cassandra/database/data/system/Versions/system-Versions-hc-34 (247 bytes) INFO 13:59:07,790 Opening /opt/cassandra/database/data/system/Versions/system-Versions-hc-35 (247 bytes) INFO 13:59:07,815 Opening /opt/cassandra/database/data/system/IndexInfo/system-IndexInfo-hc-32 (490 bytes) INFO 13:59:07,816 Opening /opt/cassandra/database/data/system/IndexInfo/system-IndexInfo-hc-33 (115 bytes) INFO 13:59:07,850 Opening /opt/cassandra/database/data/system/schema_keyspaces/system-schema_keyspaces-hc-9 (506 bytes) INFO 13:59:07,850 Opening /opt/cassandra/database/data/system/schema_keyspaces/system-schema_keyspaces-hc-10 (506 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-74 (148 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-72 (406 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-73 (80 bytes)ERROR 13:59:08,244 Exception encountered during startup java.lang.IllegalArgumentException: value already present: 1034 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111) at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96) at com.google.common.collect.HashBiMap.put(HashBiMap.java:84) at org.apache.cassandra.config.Schema.load(Schema.java:385) at org.apache.cassandra.config.Schema.load(Schema.java:106) at org.apache.cassandra.config.Schema.load(Schema.java:91) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:533) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) java.lang.IllegalArgumentException: value already present: 1034 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111) at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96) at com.google.common.collect.HashBiMap.put(HashBiMap.java:84) at org.apache.cassandra.config.Schema.load(Schema.java:385) at org.apache.cassandra.config.Schema.load(Schema.java:106) at org.apache.cassandra.config.Schema.load(Schema.java:91) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:533) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) Exception encountered during startup: value already present: 1034 What can I do to get my cluster back online? Thank you Christoph Eberhardt

Dave Brosius | 16 May 20:29 2012

Re: Startup fails after updgrading from 1.0.8 to 1.1.0

tracking issue here: https://issues.apache.org/jira/browse/CASSANDRA-4251
might be related to: https://issues.apache.org/jira/browse/CASSANDRA-3794

On 05/16/2012 08:12 AM, Christoph Eberhardt wrote:
Hi there, if updgraded cassandra from 1.0.8 to 1.1.0. It seemed to work in the first place, all seemed to work fine. So I started upgrading the rest of the cluster (at the time only one other node, that is a replica). After having a several errors, I restarted the cluster and now cassandra won't even start up. Startup fails with the following error message: INFO 13:59:05,175 Logging initialized INFO 13:59:05,178 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 13:59:05,178 Heap size: 8420720640/8420720640 INFO 13:59:05,178 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.1.0.jar:bin/../lib/apache-cassandra-clientutil-1.1.0.jar:bin/../lib/apache-cassandra-thrift-1.1.0.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.4.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:bin/../lib/guava-r08.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl- 1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jline-0.9.94.jar:bin/../lib/jna.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/metrics-core-2 . 0.3.jar:bin/../lib/mx4j-tools.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.6.1.jar:bin/../lib/slf4j-log4j12-1.6.1.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.4.1.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar INFO 13:59:07,312 JNA mlockall successful INFO 13:59:07,325 Loading settings from file:/opt/cassandra/apache-cassandra-1.1.0/conf/cassandra.yaml INFO 13:59:07,419 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 13:59:07,567 Global memtable threshold is enabled at 2676MB INFO 13:59:07,654 Initializing key cache with capacity of 100 MBs. INFO 13:59:07,661 Scheduling key cache save to each 14400 seconds (going to save all keys). INFO 13:59:07,662 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO 13:59:07,664 Scheduling row cache save to each 0 seconds (going to save all keys). INFO 13:59:07,717 Opening /opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-9 (28520 bytes) INFO 13:59:07,717 Opening /opt/cassandra/database/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-10 (28520 bytes) INFO 13:59:07,746 Opening /opt/cassandra/database/data/system/NodeIdInfo/system-NodeIdInfo-hc-1 (187 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-3 (1892 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-1 (1892 bytes) INFO 13:59:07,777 Opening /opt/cassandra/database/data/system/schema_columns/system-schema_columns-hc-2 (1892 bytes) INFO 13:59:07,790 Opening /opt/cassandra/database/data/system/Versions/system-Versions-hc-34 (247 bytes) INFO 13:59:07,790 Opening /opt/cassandra/database/data/system/Versions/system-Versions-hc-35 (247 bytes) INFO 13:59:07,815 Opening /opt/cassandra/database/data/system/IndexInfo/system-IndexInfo-hc-32 (490 bytes) INFO 13:59:07,816 Opening /opt/cassandra/database/data/system/IndexInfo/system-IndexInfo-hc-33 (115 bytes) INFO 13:59:07,850 Opening /opt/cassandra/database/data/system/schema_keyspaces/system-schema_keyspaces-hc-9 (506 bytes) INFO 13:59:07,850 Opening /opt/cassandra/database/data/system/schema_keyspaces/system-schema_keyspaces-hc-10 (506 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-74 (148 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-72 (406 bytes) INFO 13:59:07,867 Opening /opt/cassandra/database/data/system/LocationInfo/system-LocationInfo-hc-73 (80 bytes)ERROR 13:59:08,244 Exception encountered during startup java.lang.IllegalArgumentException: value already present: 1034 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111) at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96) at com.google.common.collect.HashBiMap.put(HashBiMap.java:84) at org.apache.cassandra.config.Schema.load(Schema.java:385) at org.apache.cassandra.config.Schema.load(Schema.java:106) at org.apache.cassandra.config.Schema.load(Schema.java:91) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:533) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) java.lang.IllegalArgumentException: value already present: 1034 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111) at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96) at com.google.common.collect.HashBiMap.put(HashBiMap.java:84) at org.apache.cassandra.config.Schema.load(Schema.java:385) at org.apache.cassandra.config.Schema.load(Schema.java:106) at org.apache.cassandra.config.Schema.load(Schema.java:91) at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:533) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) Exception encountered during startup: value already present: 1034 What can I do to get my cluster back online? Thank you Christoph Eberhardt


Gmane