Initializing a multiple node cluster (multiple datacenters)

A deployment scenario for a DataStax Distribution of Apache Cassandra 3.11 cluster with multiple datacenters.

1. Suppose you install Cassandra on these nodes:

   node0 10.168.66.41 (seed1)
                           node1 10.176.43.66
                           node2 10.168.247.41
                           node3 10.176.170.59 (seed2)
                           node4 10.169.61.170
                           node5 10.169.30.138

   Note: It is a best practice to have more than one seed node per datacenter.
2. If you have a firewall running in your cluster, you must open certain ports for communication between the nodes. See Configuring firewall port access.
3. If Cassandra is running, you must stop the server and clear the data:

   Doing this removes the default cluster_name (Test Cluster) from the system table. All nodes must use the same cluster name.

   1. Stop Cassandra:

      ps auwx | grep cassandra #Stops DDAC
      $ sudo  kill pid

   2. Clear the data:

      sudo rm -rf /var/lib/cassandra/data/data/system/*

4. Set the properties in the cassandra.yaml file for each node:
   Note: After making any changes in the cassandra.yaml file, you must restart the node for the changes to take effect.

   Properties to set:

   • cluster_name:
   • num_tokens: recommended value: 256
   • -seeds: internal IP address of each seed node

     In new clusters. Seed nodes don't perform bootstrap (the process of a new node joining an existing cluster.)

   • listen_address:

     If the node is a seed node, this address must match an IP address in the seeds list. Otherwise, gossip communication fails because it doesn't know that it is a seed.

     If not set, Cassandra asks the system for the local address, the one associated with its hostname. In some cases, Cassandra doesn't produce the correct address and you must specify the listen_address.

   • rpc_address:listen address for client connections
   • endpoint_snitch: name of snitch (See endpoint_snitch.) If you are changing snitches, see Switching snitches.
   • auto_bootstrap: false (Add this setting only when initializing a clean node with no data.)

   Note: If the nodes in the cluster are identical in terms of disk layout, shared libraries, and so on, you can use the same cassandra.yaml file on all of them.

   Example:

   cluster_name: 'MyCassandraCluster'
       num_tokens: 256
       seed_provider:
       - class_name: org.apache.cassandra.locator.SimpleSeedProvider
       parameters:
       - seeds:  "10.168.66.41,10.176.170.59"
       listen_address:
       endpoint_snitch: GossipingPropertyFileSnitch

   Note: Include at least one node from each datacenter in the seeds list.
5. In the cassandra-rackdc.properties file, assign the datacenter and rack names you determined in the Prerequisites. For example:

   Nodes 0 to 2

   ## Indicate the rack and dc for this node
       dc=DC1
       rack=RAC1

   Nodes 3 to 5

   ## Indicate the rack and dc for this node
       dc=DC2
       rack=RAC1

6. The GossipingPropertyFileSnitch always loads cassandra-topology.properties when that file is present. Remove the file from each node on any new cluster or any cluster migrated from the PropertyFileSnitch.
7. After you have installed and configured Cassandra on all nodes, DataStax recommends starting the seed nodes one at a time, and then starting the rest of the nodes.
   Note: If the node has restarted because of automatic restart, you must first stop the node and clear the data directories, as described above.

   cd install_location 
   $ bin/cassandra #Starts DDAC

8. To check that the ring is up and running, run:

   cd install_location 

   bin/nodetool status

DC: Cassandra
==============================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address         Load        Tokens    Owns    Host ID             Rack
UN 110.82.155.0    21.33 KB    256       50.2%   a9fa31c7-f3c0-...   RAC1
UN 110.82.155.1    21.33 KB    256       49.8%   f5bb416c-db51-...   RAC1

DC: Cassandra2
==============================================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address         Load        Tokens    Owns    Host ID             Rack
UN 110.54.125.1    21.33 KB    256       16.7%   b836748f-c94f-...   RAC2
UN 110.55.120.1    21.33 KB    256       16.7%   b354798g-c94f-...   RAC2