Package

com.datastax.spark.connector

rdd

Permalink

package rdd

Contains com.datastax.spark.connector.rdd.CassandraTableScanRDD class that is the main entry point for analyzing Cassandra data from Spark.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. rdd
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. class CassandraCoGroupedRDD[T] extends RDD[Seq[Seq[T]]]

    Permalink

    A RDD which pulls from provided separate CassandraTableScanRDDs which share partition keys type and keyspaces.

    A RDD which pulls from provided separate CassandraTableScanRDDs which share partition keys type and keyspaces. These tables will be joined on READ using a merge iterator. As long as we join on the token of the partition key the two iterators should be read in order. Note: this implementation do not restrict partition keys has the same names, but they should have the same types

  2. class CassandraJoinRDD[L, R] extends CassandraRDD[(L, R)] with CassandraTableRowReaderProvider[R] with AbstractCassandraJoin[L, R]

    Permalink

    An RDD that will do a selecting join between left RDD and the specified Cassandra Table This will perform individual selects to retrieve the rows from Cassandra and will take advantage of RDDs that have been partitioned with the com.datastax.spark.connector.rdd.partitioner.ReplicaPartitioner

    An RDD that will do a selecting join between left RDD and the specified Cassandra Table This will perform individual selects to retrieve the rows from Cassandra and will take advantage of RDDs that have been partitioned with the com.datastax.spark.connector.rdd.partitioner.ReplicaPartitioner

    L

    item type on the left side of the join (any RDD)

    R

    item type on the right side of the join (fetched from Cassandra)

  3. class CassandraLeftJoinRDD[L, R] extends CassandraRDD[(L, Option[R])] with CassandraTableRowReaderProvider[R] with AbstractCassandraJoin[L, Option[R]]

    Permalink

    An RDD that will do a selecting join between left RDD and the specified Cassandra Table This will perform individual selects to retrieve the rows from Cassandra and will take advantage of RDDs that have been partitioned with the com.datastax.spark.connector.rdd.partitioner.ReplicaPartitioner

    An RDD that will do a selecting join between left RDD and the specified Cassandra Table This will perform individual selects to retrieve the rows from Cassandra and will take advantage of RDDs that have been partitioned with the com.datastax.spark.connector.rdd.partitioner.ReplicaPartitioner

    L

    item type on the left side of the join (any RDD)

    R

    item type on the right side of the join (fetched from Cassandra)

  4. sealed trait CassandraLimit extends AnyRef

    Permalink
  5. class CassandraMergeJoinRDD[L, R] extends RDD[(Seq[L], Seq[R])]

    Permalink

    A RDD which pulls from two separate CassandraTableScanRDDs which share partition keys and keyspaces.

    A RDD which pulls from two separate CassandraTableScanRDDs which share partition keys and keyspaces. These tables will be joined on READ using a merge iterator. As long as we join on the token of the partition key the two iterators should be read in order.

  6. case class CassandraPartitionLimit(rowsNumber: Long) extends CassandraLimit with Product with Serializable

    Permalink
  7. abstract class CassandraRDD[R] extends RDD[R]

    Permalink
  8. trait CassandraTableRowReaderProvider[R] extends AnyRef

    Permalink

    Used to get a RowReader of type [R] for transforming the rows of a particular Cassandra table into scala objects.

    Used to get a RowReader of type [R] for transforming the rows of a particular Cassandra table into scala objects. Performs necessary checking of the schema and output class to make sure they are compatible.

    See also

    CassandraJoinRDD

    CassandraTableScanRDD

  9. class CassandraTableScanRDD[R] extends CassandraRDD[R] with CassandraTableRowReaderProvider[R] with SplitSizeEstimator[R]

    Permalink

    RDD representing a Table Scan of A Cassandra table.

    RDD representing a Table Scan of A Cassandra table.

    This class is the main entry point for analyzing data in Cassandra database with Spark. Obtain objects of this class by calling com.datastax.spark.connector.SparkContextFunctions.cassandraTable.

    Configuration properties should be passed in the SparkConf configuration of SparkContext. CassandraRDD needs to open connection to Cassandra, therefore it requires appropriate connection property values to be present in SparkConf. For the list of required and available properties, see CassandraConnector.

    CassandraRDD divides the data set into smaller partitions, processed locally on every cluster node. A data partition consists of one or more contiguous token ranges. To reduce the number of roundtrips to Cassandra, every partition is fetched in batches.

    The following properties control the number of partitions and the fetch size: - spark.cassandra.input.split.sizeInMB: approx amount of data to be fetched into a single Spark partition, default 512 MB - spark.cassandra.input.fetch.sizeInRows: number of CQL rows fetched per roundtrip, default 1000

    A CassandraRDD object gets serialized and sent to every Spark Executor, which then calls the compute method to fetch the data on every node. The getPreferredLocations method tells Spark the preferred nodes to fetch a partition from, so that the data for the partition are at the same node the task was sent to. If Cassandra nodes are collocated with Spark nodes, the queries are always sent to the Cassandra process running on the same node as the Spark Executor process, hence data are not transferred between nodes. If a Cassandra node fails or gets overloaded during read, the queries are retried to a different node.

    By default, reads are performed at ConsistencyLevel.LOCAL_ONE in order to leverage data-locality and minimize network traffic. This read consistency level is controlled by the spark.cassandra.input.consistency.level property.

  10. sealed trait ClusteringOrder extends Serializable

    Permalink
  11. case class CqlWhereClause(predicates: Seq[String], values: Seq[Any]) extends Product with Serializable

    Permalink

    Represents a logical conjunction of CQL predicates.

    Represents a logical conjunction of CQL predicates. Each predicate can have placeholders denoted by '?' which get substituted by values from the values array. The number of placeholders must match the size of the values array.

  12. class DseGraphPartitioner[V, T <: Token[V]] extends Partitioner with Logging

    Permalink

    A custom partitoner specifically for RDDs made for DseGraph.

    A custom partitoner specifically for RDDs made for DseGraph.

    The general idea is

    For a vertex Determine the ~label property Determine which RDD represents the data in that label Determine the C* token of the vertex given it's label Determine which partition in the found RDD would contain that Token Determine the offset of that RDD's partitions in the UnionRDD Return the partition index added to offset

  13. class DseGraphUnionedRDD[R] extends UnionRDD[R]

    Permalink

    An extension of UnionRDD which automatically assigns a partitioner based on the way DseGraph stores and partitions vertex information.

    An extension of UnionRDD which automatically assigns a partitioner based on the way DseGraph stores and partitions vertex information. The graphLabels should map 1 to 1 with the RDDs provided in sequence. This ordering is used to develop a mapping between labels and the RDDs which represent the data for that label.

  14. class EmptyCassandraRDD[R] extends CassandraRDD[R]

    Permalink

    Represents a CassandraRDD with no rows.

    Represents a CassandraRDD with no rows. This RDD does not load any data from Cassandra and doesn't require for the table to exist.

  15. class MapRowWriter extends RowWriter[Map[String, AnyRef]]

    Permalink
  16. case class ReadConf(splitCount: Option[Int] = None, splitSizeInMB: Int = ReadConf.SplitSizeInMBParam.default, fetchSizeInRows: Int = ..., consistencyLevel: ConsistencyLevel = ..., taskMetricsEnabled: Boolean = ReadConf.TaskMetricParam.default, throughputMiBPS: Option[Double] = None, readsPerSec: Option[Int] = ReadConf.ReadsPerSecParam.default, parallelismLevel: Int = ..., executeAs: Option[String] = None) extends Product with Serializable

    Permalink

    Read settings for RDD

    Read settings for RDD

    splitCount

    number of partitions to divide the data into; unset by default

    splitSizeInMB

    size of Cassandra data to be read in a single Spark task; determines the number of partitions, but ignored if splitCount is set

    fetchSizeInRows

    number of CQL rows to fetch in a single round-trip to Cassandra

    consistencyLevel

    consistency level for reads, default LOCAL_ONE; higher consistency level will disable data-locality

    taskMetricsEnabled

    whether or not enable task metrics updates (requires Spark 1.2+)

    readsPerSec

    maximum read throughput allowed per single core in requests/s while joining an RDD with C* table (joinWithCassandraTable operation) also used by enterprise integrations

  17. case class SparkPartitionLimit(rowsNumber: Long) extends CassandraLimit with Product with Serializable

    Permalink
  18. trait ValidRDDType[T] extends AnyRef

    Permalink
    Annotations
    @implicitNotFound( ... )

Value Members

  1. object CassandraCoGroupedRDD extends Serializable

    Permalink
  2. object CassandraLimit

    Permalink
  3. object CassandraRDD extends Serializable

    Permalink
  4. object CassandraTableScanRDD extends Serializable

    Permalink
  5. object ClusteringOrder extends Serializable

    Permalink
  6. object CqlWhereClause extends Serializable

    Permalink
  7. object DseGraphUnionedRDD extends Serializable

    Permalink

    A Java Friendly api for DseGraphUnionedRDD to make it easier to call from VertexInputRDD

  8. object ReadConf extends Logging with Serializable

    Permalink
  9. object ValidRDDType

    Permalink
  10. package partitioner

    Permalink

    Provides components for partitioning a Cassandra table into smaller parts of appropriate size.

    Provides components for partitioning a Cassandra table into smaller parts of appropriate size. Each partition can be processed locally on at least one cluster node.

  11. package reader

    Permalink

    Provides components for reading data rows from Cassandra and converting them to objects of desired type.

    Provides components for reading data rows from Cassandra and converting them to objects of desired type. Additionally provides a generic CassandraRow class which can represent any row.

Inherited from AnyRef

Inherited from Any

Ungrouped