Hadoop

Hadoop Status Methods URL
Get the current status of the Hadoop cluster. GET /{cluster_id}/hadoop/status
Get a list of submitted jobs and their status. GET /{cluster_id}/hadoop/jobs
Get the proxy port for the Hadoop UI GET /{cluster_id}/hadoop/jt/port

Hadoop Status

GET /{cluster_id}/hadoop/status

Retrieve information about the status of the Hadoop cluster. Returns a JSON dictionary of cluster attributes and their current value.

Example:

curl http://127.0.0.1:8888/Test_Cluster/hadoop/status

Output:

{
    "activeTrackerNames": [
        "tracker_dse1:dse1/127.0.0.1:47333"
    ],
    "blacklisted_tracker_names": [],
    "has_recovered": false,
    "has_restarted": false,
    "hostname": "dse1",
    "http_port": 50030,
    "identifier": "201301301843",
    "mapTasks": 0,
    "max_map_tasks": 2,
    "max_memory": 4125097984,
    "max_reduce_tasks": 2,
    "num_active_trackers": 1,
    "num_blacklisted_trackers": 0,
    "num_excluded_nodes": 0,
    "reduce_tasks": 0,
    "start_time": 1359592991647,
    "state": 1,
    "task_tracker_expiry_interval": 600000,
    "total_submissions": 5,
    "used_memory": 4125097984
}

Job Details

GET /{cluster_id}/hadoop/jobs

Returns a JSON list of submitted jobs, each of which is a dictionary of job attributes.

Example:

curl http://127.0.0.1:8888/Test_Cluster/hadoop/jobs

Output:

[
    {
        "desired_maps": 1,
        "desired_reduces": 1,
        "duration": 18951,
        "finish_time": 0,
        "finished_maps": 1,
        "finished_reduces": 0,
        "job_id": {
            "as_string": "job_201301301843_0001",
            "job_id": 1,
            "job_tracker_id": "201301301843"
        },
        "launch_time": 1359593329324,
        "priority": 2,
        "profile": {
            "job_file": "cfs:/tmp/hadoop-joe/mapred/staging/joe/.staging/job_201301301843_0001/job.xml",
            "name": "INSERT OVERWRITE TABLE 10da...b.column_name)(Stage-1)",
            "queue_name": "default",
            "user": "joe"
        },
        "start_time": 1359593329128,
        "status": {
            "cleanup_progress": 0.0,
            "map_progress": 1.0,
            "reduce_progress": 0.0,
            "run_state": 1,
            "scheduling_info": "NA",
            "setup_progress": 1.0
        }
    }
]

Proxy Port

GET /{cluster_id}/hadoop/jt/port

Retrieve the proxy port used by OpsCenter to expose the Hadoop HTTP UI

Example:

curl http://127.0.0.1:8888/Test_Cluster/hadoop/jt/port

Output:

50031