Class

com.datastax.spark.connector

RDDFunctions

Related Doc: package connector

Permalink

class RDDFunctions[T] extends WritableToCassandra[T] with Serializable

Provides Cassandra-specific methods on RDD

Linear Supertypes
Serializable, Serializable, WritableToCassandra[T], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RDDFunctions
  2. Serializable
  3. Serializable
  4. WritableToCassandra
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new RDDFunctions(rdd: RDD[T])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. def deleteFromCassandra(keyspaceName: String, tableName: String, deleteColumns: ColumnSelector = SomeColumns(), keyColumns: ColumnSelector = PrimaryKeyColumns, writeConf: WriteConf = ...)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), rwf: RowWriterFactory[T]): Unit

    Permalink

    Delete data from Cassandra table, using data from the RDD as primary keys.

    Delete data from Cassandra table, using data from the RDD as primary keys. Uses the specified column names.

    keyspaceName

    the name of the Keyspace to use

    tableName

    the name of the Table to use

    deleteColumns

    The list of column names to delete, empty ColumnSelector means full row.

    keyColumns

    Primary key columns selector, Optional. All RDD primary columns columns will be checked by default

    writeConf

    additional configuration object allowing to set consistency level, batch size, etc.

    Definition Classes
    RDDFunctionsWritableToCassandra
    See also

    com.datastax.spark.connector.writer.WritableToCassandra

  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  11. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  12. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  13. def joinWithCassandraTable[R](keyspaceName: String, tableName: String, selectedColumns: ColumnSelector = AllColumns, joinColumns: ColumnSelector = PartitionKeyColumns)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), newType: ClassTag[R], rrf: RowReaderFactory[R], ev: ValidRDDType[R], currentType: ClassTag[T], rwf: RowWriterFactory[T]): CassandraJoinRDD[T, R]

    Permalink

    Uses the data from RDD to join with a Cassandra table without retrieving the entire table.

    Uses the data from RDD to join with a Cassandra table without retrieving the entire table. Any RDD which can be used to saveToCassandra can be used to joinWithCassandra as well as any RDD which only specifies the partition Key of a Cassandra Table. This method executes single partition requests against the Cassandra Table and accepts the functional modifiers that a normal com.datastax.spark.connector.rdd.CassandraTableScanRDD takes.

    By default this method only uses the Partition Key for joining but any combination of columns which are acceptable to C* can be used in the join. Specify columns using joinColumns as a parameter or the on() method.

    Example With Prior Repartitioning:

    val source = sc.parallelize(keys).map(x => new KVRow(x))
    val repart = source.repartitionByCassandraReplica(keyspace, tableName, 10)
    val someCass = repart.joinWithCassandraTable(keyspace, tableName)

    Example Joining on Clustering Columns:

    val source = sc.parallelize(keys).map(x => (x, x * 100))
    val someCass = source.joinWithCassandraTable(keyspace, wideTable).on(SomeColumns("key", "group"))
  14. def keyByCassandraReplica(keyspaceName: String, tableName: String, partitionKeyMapper: ColumnSelector = PartitionKeyColumns)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), currentType: ClassTag[T], rwf: RowWriterFactory[T]): RDD[(Set[InetAddress], T)]

    Permalink

    Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row.

    Key every row in the RDD by with the IP Adresses of all of the Cassandra nodes which a contain a replica of the data specified by that row. The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.

  15. def leftJoinWithCassandraTable[R](keyspaceName: String, tableName: String, selectedColumns: ColumnSelector = AllColumns, joinColumns: ColumnSelector = PartitionKeyColumns)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), newType: ClassTag[R], rrf: RowReaderFactory[R], ev: ValidRDDType[R], currentType: ClassTag[T], rwf: RowWriterFactory[T]): CassandraLeftJoinRDD[T, R]

    Permalink

    Uses the data from RDD to left join with a Cassandra table without retrieving the entire table.

    Uses the data from RDD to left join with a Cassandra table without retrieving the entire table. Any RDD which can be used to saveToCassandra can be used to leftJoinWithCassandra as well as any RDD which only specifies the partition Key of a Cassandra Table. This method executes single partition requests against the Cassandra Table and accepts the functional modifiers that a normal com.datastax.spark.connector.rdd.CassandraTableScanRDD takes.

    By default this method only uses the Partition Key for joining but any combination of columns which are acceptable to C* can be used in the join. Specify columns using joinColumns as a parameter or the on() method.

    Example With Prior Repartitioning:

    val source = sc.parallelize(keys).map(x => new KVRow(x))
    val repart = source.repartitionByCassandraReplica(keyspace, tableName, 10)
    val someCass = repart.leftJoinWithCassandraTable(keyspace, tableName)

    Example Joining on Clustering Columns:

    val source = sc.parallelize(keys).map(x => (x, x * 100))
    val someCass = source.leftJoinWithCassandraTable(keyspace, wideTable).on(SomeColumns("key", "group"))
  16. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. def repartitionByCassandraReplica(keyspaceName: String, tableName: String, partitionsPerHost: Int = 10, partitionKeyMapper: ColumnSelector = PartitionKeyColumns)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), currentType: ClassTag[T], rwf: RowWriterFactory[T]): CassandraPartitionedRDD[T]

    Permalink

    Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName.

    Repartitions the data (via a shuffle) based upon the replication of the given keyspaceName and tableName. Calling this method before using joinWithCassandraTable will ensure that requests will be coordinator local. partitionsPerHost Controls the number of Spark Partitions that will be created in this repartitioning event. The calling RDD must have rows that can be converted into the partition key of the given Cassandra Table.

  20. def saveAsCassandraTable(keyspaceName: String, tableName: String, columns: ColumnSelector = AllColumns, writeConf: WriteConf = ...)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), rwf: RowWriterFactory[T], columnMapper: ColumnMapper[T]): Unit

    Permalink

    Saves the data from RDD to a new table with definition taken from the ColumnMapper for this class.

    Saves the data from RDD to a new table with definition taken from the ColumnMapper for this class.

    keyspaceName

    keyspace where to create a new table

    tableName

    name of the table to create; the table must not exist

    columns

    Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.

    writeConf

    additional configuration object allowing to set consistency level, batch size, etc.

    connector

    optional, implicit connector to Cassandra

    rwf

    factory for obtaining the row writer to be used to extract column values from items of the RDD

    columnMapper

    a column mapper determining the definition of the table

  21. def saveAsCassandraTableEx(table: TableDef, columns: ColumnSelector = AllColumns, writeConf: WriteConf = ...)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), rwf: RowWriterFactory[T]): Unit

    Permalink

    Saves the data from RDD to a new table defined by the given TableDef.

    Saves the data from RDD to a new table defined by the given TableDef.

    First it creates a new table with all columns from the TableDef and then it saves RDD content in the same way as saveToCassandra. The table must not exist prior to this call.

    table

    table definition used to create a new table

    columns

    Selects the columns to save data to. Uses only the unique column names, and you must select at least all primary key columns. All other fields are discarded. Non-selected property/column names are left unchanged. This parameter does not affect table creation.

    writeConf

    additional configuration object allowing to set consistency level, batch size, etc.

    connector

    optional, implicit connector to Cassandra

    rwf

    factory for obtaining the row writer to be used to extract column values from items of the RDD

  22. def saveToCassandra(keyspaceName: String, tableName: String, columns: ColumnSelector = AllColumns, writeConf: WriteConf = ...)(implicit connector: CassandraConnector = CassandraConnector(sparkContext), rwf: RowWriterFactory[T]): Unit

    Permalink

    Saves the data from RDD to a Cassandra table.

    Saves the data from RDD to a Cassandra table. Uses the specified column names.

    keyspaceName

    the name of the Keyspace to use

    tableName

    the name of the Table to use

    writeConf

    additional configuration object allowing to set consistency level, batch size, etc.

    Definition Classes
    RDDFunctionsWritableToCassandra
    See also

    com.datastax.spark.connector.writer.WritableToCassandra

  23. def spanBy[U](f: (T) ⇒ U): RDD[(U, Iterable[T])]

    Permalink

    Applies a function to each item, and groups consecutive items having the same value together.

    Applies a function to each item, and groups consecutive items having the same value together. Contrary to groupBy, items from the same group must be already next to each other in the original collection. Works locally on each partition, so items from different partitions will never be placed in the same group.

  24. val sparkContext: SparkContext

    Permalink
    Definition Classes
    RDDFunctionsWritableToCassandra
  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  26. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  27. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from WritableToCassandra[T]

Inherited from AnyRef

Inherited from Any

Ungrouped