Unable To Start DSE Post 5.1 Upgrade
ERROR [main] 2017-11-20 14:33:58,931 DseDaemon.java:494 - Unable to start DSE server.
com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.plugin.DseFsPlugin
Caused by: org.apache.cassandra.exceptions.ConfigurationException: DSEFS does not support authentication method configured with org.apache.cassandra.auth.PasswordAuthenticator. DSEFS supports INTERNAL, LDAP and DIGEST authentication schemes configured with DseAuthenticator.
As you can see, the default Authenticator and Authorizer are being deprecated and now changed to following.
authenticator: com.datastax.bdp.cassandra.auth.DseAuthenticator
authorizer: com.datastax.bdp.cassandra.auth.DseAuthorizer
role_manager: com.datastax.bdp.cassandra.auth.DseRoleManager
In the dse.yaml file, configure the corresponding options:
Configure the DSE Authenticator by uncommenting the authentication_options and changing the settings.
FROM -
# authentication_options:
# enabled: false
# default_scheme: internal
TO -
enabled: true
default_scheme: internal
Configure the DSE Role Manager by uncommenting role_management_options and setting the mode
role_management_options:
mode: internal
Configure the DSE Authorizer by uncommenting the authorization_options and changing the settings.
authorization_options:
enabled: true
# transitional_mode: normal
# allow_row_level_security: false
Once done, run nodetool upgradesstables, if you have not already ran it.
[root@hqidlinfdb36 cassandra]# nodetool upgradesstables
WARN 14:50:54,675 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 243. You can override this in cassandra.yaml
WARN 14:50:54,681 memtable_cleanup_threshold has been deprecated and should be removed from cassandra.yaml
You must configure the replication factors appropriate for using DSE Security in production environments. The keyspaces that require an increased replication factor are:
system_authdse_security
Change the system_auth keyspace RF:
ALTER KEYSPACE system_auth
WITH REPLICATION= {'class' : 'NetworkTopologyStrategy',
'data_center_name' : N,
'data_center_name' : N};
Example - ALTER KEYSPACE system_auth WITH REPLICATION =
{'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};
Change the dse_security keyspace RF:
ALTER KEYSPACE "dse_security"
WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};
Run the nodetool repair on the security keyspaces.
nodetool repair --full system_auth
nodetool repair --full dse_security
After changing the replication strategy, you must run nodetool repair with the --full option. when running full repair, you may see following warning where one has to modify the RF for dse_leases key space.
insufficient replication (you created a new DC and didn't ALTER KEYSPACE dse_leases) and the duration (30000) being different (you have to disable/delete/recreate the lease to change the duration).
java.io.IOException: No live replicas for lease Leader/master/5.1.dc1 in table dse_leases.leases Nodes [/10.22.102.27] are all down/still starting.
ALTER KEYSPACE dse_leases WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'} AND durable_writes = true;
Once this is done, you are now running successfully on 5.1 version.
