Using Hive

DataStax Enterprise includes a Cassandra-enabled Hive MapReduce client. Hive is a data warehouse system for Hadoop that projects a relational structure onto data stored in Hadoop-compatible file systems. You query the data using a SQL-like language called HiveQL.

You can start the Hive client on any analytics node and run MapReduce queries directly on data already stored in Cassandra. Using the DataStax Enterprise ODBC driver for Hive, a JDBC compliant user interface can connect to Hive from the Hive server.

Why Hive

By using Hive, you typically eliminate boilerplate MapReduce code and enjoy productivity gains. The large base of SQL users can master HiveQL quickly. Hive has a large set of standard functions, such as mathematical and string functions. You can use Hive for queries that Cassandra as a NoSQL database does not support, such as joins. DataStax Enterprise support of Hive facilitates the migration of data to DataStax Enterprise from a Hive warehouse. Hive capabilities are extensible through Hive user-defined function (UDF), Hive user-defined function (UDF) which DataStax Enterprise 3.2 supports.

Typical uses for Hive are:

Reporting
User engagement and impression click count applications
Ad hoc analysis
Machine learning
Advertising optimization

Hive in DataStax Enterprise

DataStax Enterprise analytics nodes store Hive table structures in the CassandraFS instead of in a Hadoop Distributed File System (HDFS). You layer a Hive table definition onto a directory in the file system or onto a Cassandra CQL table. The Hive table definition describes the layout of the data and is stored in the HiveMetaStore keyspace. DataStax Enterprise implements the Hive metastore as the HiveMetaStore keyspace within Cassandra. Unlike open source Hive, there is no need to run the metastore as a standalone database to support multiple users.

The consistency level of Hadoop nodes is ONE by default, but when processing Hive queries, if DataStax Enterprise (3.1.5 and later) can guarantee that a replicas are in the same data center, the consistency level of LOCAL_ONE is used.

You can map an existing CQL table to Hive a table by creating an external table. The external table data source is external to Hive, located in CQL. When you drop a Hive external table, only the table metadata stored in the HiveMetaStore keyspace is removed. The data persists in CQL.

Instead of an external table, you can use a Hive managed table. Hive manages storing and deleting the data in this type of table. The data source can be a flat file that you put on the CassandraFS (using a hadoop -fs command) or the file can be elsewhere, such as on an operating system file system. To load the file, use the LOAD [LOCAL] DATA INPATH, INSERT INTO, or INSERT OVERWRITE Hive commands.

Hive metastore configuration

The HiveMetaStore in DataStax Enterprise supports multiple users and requires no configuration except increasing the default replication factor of the keyspace. The default replication for system keyspaces is 1. This replication factor is suitable for development and testing, not for a production environment. To avoid production problems, alter the replication factor of these system keyspaces from 1 to a higher number.

HiveMetaStore
cfs
cfs_archive keyspaces

To prevent missing data problems or data unavailable exceptions after altering keyspaces that contain any data, run nodetool repair.

New Hive features

DataStax Enterprise 3.2 improves Hive data mapping and now supports most CQL and Cassandra internal data types. Another new addition is a Hive user-defined function (UDF) for working with unsupported types, such as blob. DataStax Enterprise provides this UDF:

org.apache.hadoop.hive.cassandra.ql.udf.UDFStringToCassandraBinary

This UDF converts from Hive Strings to native Cassandra types. Due to limitations in Hive, the UDF can be used only to convert Hive Strings to string primitives, not collections that are arrays and maps of strings. It is not possible to use the UDF to convert, for example, an array of strings representing inet addresses to an array of InetAddress columns in Cassandra.

CQL collections and the CQL composite partition keys are now supported.

The Hive examples in this documentation show how to use new features:

There are also examples of using external and managed tables.