Commands Guide Manual

commands_manual

User Manual:

Open the PDF directly: View PDF .
Page Count: 12

Download
Open PDF In Browser	View PDF

Commands Guide
Table of contents
1 Overview............................................................................................................................ 3
1.1 Generic Options............................................................................................................ 3
2 User Commands ................................................................................................................4
2.1 archive ..........................................................................................................................4
2.2 distcp ............................................................................................................................4
2.3 fs .................................................................................................................................. 4
2.4 fsck ...............................................................................................................................4
2.5 fetchdt .......................................................................................................................... 5
2.6 jar ................................................................................................................................. 5
2.7 job ................................................................................................................................ 5
2.8 pipes .............................................................................................................................6
2.9 queue ............................................................................................................................7
2.10 version ........................................................................................................................7
2.11 CLASSNAME ........................................................................................................... 8
2.12 classpath .....................................................................................................................8
3 Administration Commands ............................................................................................... 8
3.1 balancer ........................................................................................................................8
3.2 daemonlog ....................................................................................................................8
3.3 datanode........................................................................................................................ 9
3.4 dfsadmin ...................................................................................................................... 9
3.5 mradmin...................................................................................................................... 10
3.6 jobtracker ................................................................................................................... 11
3.7 namenode ................................................................................................................... 11
Copyright © 2008 The Apache Software Foundation. All rights reserved.

Commands Guide

3.8 secondarynamenode ................................................................................................... 12
3.9 tasktracker ..................................................................................................................12

Page 2

Commands Guide

1 Overview
All hadoop commands are invoked by the bin/hadoop script. Running the hadoop script
without any arguments prints the description for all commands.
Usage: hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS]
[COMMAND_OPTIONS]
Hadoop has an option parsing framework that employs parsing generic options as well as
running classes.
COMMAND_OPTION

Description

--config confdir

Overwrites the default Configuration directory.
Default is ${HADOOP_HOME}/conf.

GENERIC_OPTIONS

The common set of options supported by multiple
commands.

COMMAND
COMMAND_OPTIONS

Various commands with their options are described
in the following sections. The commands have been
grouped into User Commands and Administration
Commands.

1.1 Generic Options

The following options are supported by dfsadmin, fs, fsck, job and fetchdt. Applications
should implement Tool to support GenericOptions.
GENERIC_OPTION

Description

-conf

Specify an application configuration file.

-D

Use value for given property.

-fs

Specify a namenode.

-jt

Specify a job tracker. Applies only to job.

-files

Specify comma separated files to be copied to the
map reduce cluster. Applies only to job.

-libjars

Specify comma separated jar files to include in the
classpath. Applies only to job.

-archives

Specify comma separated archives to be unarchived
on the compute machines. Applies only to job.

Page 3

Commands Guide

2 User Commands
Commands useful for users of a hadoop cluster.
2.1 archive

Creates a hadoop archive. More information can be found at Hadoop Archives.
Usage: hadoop archive -archiveName NAME *
COMMAND_OPTION

Description

-archiveName NAME

Name of the archive to be created.

src

Filesystem pathnames which work as usual with
regular expressions.

dest

Destination directory which would contain the
archive.

2.2 distcp

Copy file or directories recursively. More information can be found at Hadoop DistCp Guide.
Usage: hadoop distcp
COMMAND_OPTION

Description

srcurl

Source Url

desturl

Destination Url

2.3 fs

Usage: hadoop fs [GENERIC_OPTIONS] [COMMAND_OPTIONS]
Runs a generic filesystem user client.
The various COMMAND_OPTIONS can be found at File System Shell Guide.
2.4 fsck

Runs a HDFS filesystem checking utility. See Fsck for more info.
Usage: hadoop fsck [GENERIC_OPTIONS] [-move | -delete |
-openforwrite] [-files [-blocks [-locations | -racks]]]
COMMAND_OPTION

Description
Start checking from this path.

Page 4

Commands Guide

COMMAND_OPTION

Description

-move

Move corrupted files to /lost+found

-delete

Delete corrupted files.

-openforwrite

Print out files opened for write.

-files

Print out files being checked.

-blocks

Print out block report.

-locations

Print out locations for every block.

-racks

Print out network topology for data-node locations.

2.5 fetchdt

Gets Delegation Token from a NameNode. See fetchdt for more info.
Usage: hadoop fetchdt [GENERIC_OPTIONS] [--webservice
]
COMMAND_OPTION

Description

File name to store the token into.

--webservice

use http protocol instead of RPC

2.6 jar

Runs a jar file. Users can bundle their Map Reduce code in a jar file and execute it using this
command.
Usage: hadoop jar [mainClass] args...
The streaming jobs are run via this command. Examples can be referred from Streaming
examples
Word count example is also run using jar command. It can be referred from Wordcount
example
2.7 job

Page 5

Commands Guide

[all]] | [-kill-task ] | [-fail-task ] | [set-priority ]
COMMAND_OPTION

Description

-submit

Submits the job.

-status

Prints the map and reduce completion percentage and
all job counters.

-counter

Prints the counter value.

-kill

Kills the job.

-events <#of-events>

Prints the events' details received by jobtracker for
the given range.

-history [all]

-history prints job details, failed and
killed tip details. More details about the job such as
successful tasks and task attempts made for each task
can be viewed by specifying the [all] option.

-list [all]

-list all displays all jobs. -list displays only jobs
which are yet to complete.

-kill-task

Kills the task. Killed tasks are NOT counted against
failed attempts.

-fail-task

Fails the task. Failed tasks are counted against failed
attempts.

-set-priority

Changes the priority of the job. Allowed priority
values are VERY_HIGH, HIGH, NORMAL, LOW,
VERY_LOW

2.8 pipes

Runs a pipes job.
Usage: hadoop pipes [-conf ] [-jobconf ,
, ...] [-input ] [-output ] [-jar ] [-inputformat ] [-map ] [-partitioner
] [-reduce ] [-writer ] [-program
] [-reduces ]
COMMAND_OPTION
-conf

Description
Configuration for job

Page 6

Commands Guide

COMMAND_OPTION

Description

-jobconf ,
, ...

Add/override configuration for job

-input

Input directory

-output

Output directory

-jar

Jar filename

-inputformat

InputFormat class

-map

Java Map class

-partitioner

Java Partitioner

-reduce

Java Reduce class

-writer

Java RecordWriter

-program

Executable URI

-reduces

Number of reduces

2.9 queue

command to interact and view Job Queue information
Usage : hadoop queue [-list] | [-info [showJobs]] | [-showacls]
COMMAND_OPTION

Description

-list

Gets list of Job Queues configured in the system.
Along with scheduling information associated with
the job queues.

-info [-showJobs]

Displays the job queue information and associated
scheduling information of particular job queue. If showJobs options is present a list of jobs submitted to
the particular job queue is displayed.

-showacls

Displays the queue name and associated queue
operations allowed for the current user. The list
consists of only those queues to which the user has
access.

2.10 version

Prints the version.
Copyright © 2008 The Apache Software Foundation. All rights reserved.

Page 7

Commands Guide

Usage: hadoop version
2.11 CLASSNAME

hadoop script can be used to invoke any class.
Usage: hadoop CLASSNAME
Runs the class named CLASSNAME.
2.12 classpath

Prints the class path needed to get the Hadoop jar and the required libraries.
Usage: hadoop classpath
3 Administration Commands
Commands useful for administrators of a hadoop cluster.
3.1 balancer

Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the
rebalancing process. See Rebalancer for more details.
Usage: hadoop balancer [-threshold ]
COMMAND_OPTION
-threshold

Description
Percentage of disk capacity. This overwrites the
default threshold.

3.2 daemonlog

Get/Set the log level for each daemon.
Usage: hadoop daemonlog -getlevel
Usage: hadoop daemonlog -setlevel
COMMAND_OPTION
-getlevel

Description
Prints the log level of the daemon running at
. This command internally connects to
http:///logLevel?log=

-setlevel Sets the log level of the daemon running at
. This command internally connects to
http:///logLevel?log=

Page 8

Commands Guide

3.3 datanode

Runs a HDFS datanode.
Usage: hadoop datanode [-rollback]
COMMAND_OPTION

Description
Rollsback the datanode to the previous version.
This should be used after stopping the datanode and
distributing the old hadoop version.

-rollback

3.4 dfsadmin

Runs a HDFS dfsadmin client.
Usage: hadoop dfsadmin [GENERIC_OPTIONS] [-report] [-safemode
enter | leave | get | wait] [-refreshNodes] [-finalizeUpgrade]
[-upgradeProgress status | details | force] [-metasave
filename] [-setQuota ...] [-clrQuota
...] [-help [cmd]]
COMMAND_OPTION

Description
Reports basic filesystem information and statistics.

-report

-safemode enter | leave | get | wait Safe mode maintenance command. Safe mode is a
Namenode state in which it
1. does not accept changes to the name space (readonly)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode
startup, and leaves safe mode automatically when the
configured minimum percentage of blocks satisfies
the minimum replication condition. Safe mode can
also be entered manually, but then it can only be
turned off manually as well.
-refreshNodes

Re-read the hosts and exclude files to update the
set of Datanodes that are allowed to connect to the
Namenode and those that should be decommissioned
or recommissioned.

-finalizeUpgrade

Finalize upgrade of HDFS. Datanodes delete their
previous version working directories, followed
by Namenode doing the same. This completes the
upgrade process.

Page 9

Commands Guide

COMMAND_OPTION

Description

-upgradeProgress status | details |
force

Request current distributed upgrade status, a detailed
status or force the upgrade to proceed.

-metasave filename

Save Namenode's primary data structures
to in the directory specified by
hadoop.log.dir property. will contain one
line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted

-setQuota
...

Set the quota for each directory .
The directory quota is a long integer that puts a hard
limit on the number of names in the directory tree.
Best effort for the directory, with faults reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
4. the directory would immediately exceed the new
quota.

-clrQuota ...

Clear the quota for each directory .
Best effort for the directory. with fault reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.

-help [cmd]

Displays help for the given command or all
commands if none is specified.

3.5 mradmin

Runs MR admin client
Usage: hadoop mradmin [ GENERIC_OPTIONS ] [-refreshQueueAcls]
COMMAND_OPTION
-refreshQueueAcls

Description
Refresh the queue acls used by hadoop, to check
access during submissions and administration of the
job by the user. The properties present in mapredqueue-acls.xml is reloaded by the queue
manager.

Page 10

Commands Guide

3.6 jobtracker

Runs the MapReduce job Tracker node.
Usage: hadoop jobtracker [-dumpConfiguration]
COMMAND_OPTION
-dumpConfiguration

Description
Dumps the configuration used by the JobTracker
alongwith queue configuration in JSON format into
Standard output used by the jobtracker and exits.

3.7 namenode

Runs the namenode. More info about the upgrade, rollback and finalize is at Upgrade
Rollback
Usage: hadoop namenode [-format [-force] [-nonInteractive]] |
[-upgrade] | [-rollback] | [-finalize] | [-importCheckpoint]
COMMAND_OPTION

Description

-format [-force] [-nonInteractive]

Formats the namenode. It starts the namenode,
formats it and then shuts it down. User will be
prompted for input if the name directories exist on
the local filesystem.
-nonInteractive: User will not be prompted for input
if the name directories exist in the local filesystem
and the format will fail.
-force: formats the namenode and the user will
NOT be prompted to confirm formatting of name
directories in the local filesystem. If -nonInteractive
option is specified it will be ignored.

-upgrade

Namenode should be started with upgrade option
after the distribution of new hadoop version.

-rollback

Rollsback the namenode to the previous version.
This should be used after stopping the cluster and
distributing the old hadoop version.

-finalize

Finalize will remove the previous state of the files
system. Recent upgrade will become permanent.
Rollback option will not be available anymore. After
finalization it shuts the namenode down.

-importCheckpoint

Loads image from a checkpoint directory and save
it into the current one. Checkpoint dir is read from
property fs.checkpoint.dir

Page 11

Commands Guide

3.8 secondarynamenode

Runs the HDFS secondary namenode. See Secondary Namenode for more info.
Usage: hadoop secondarynamenode [-checkpoint [force]] | [geteditsize]
COMMAND_OPTION

Description

-checkpoint [force]

Checkpoints the Secondary namenode if EditLog size
>= fs.checkpoint.size. If -force is used, checkpoint
irrespective of EditLog size.

-geteditsize

Prints the EditLog size.

3.9 tasktracker

Runs a MapReduce task Tracker node.
Usage: hadoop tasktracker

Page 12

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
Linearized                      : No
Page Count                      : 12
Profile CMM Type                : Linotronic
Profile Version                 : 2.1.0
Profile Class                   : Display Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 1998:02:09 06:49:00
Profile File Signature          : acsp
Primary Platform                : Microsoft Corporation
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : Hewlett-Packard
Device Model                    : sRGB
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 : Hewlett-Packard
Profile ID                      : 0
Profile Copyright               : Copyright (c) 1998 Hewlett-Packard Company
Profile Description             : sRGB IEC61966-2.1
Media White Point               : 0.95045 1 1.08905
Media Black Point               : 0 0 0
Red Matrix Column               : 0.43607 0.22249 0.01392
Green Matrix Column             : 0.38515 0.71687 0.09708
Blue Matrix Column              : 0.14307 0.06061 0.7141
Device Mfg Desc                 : IEC http://www.iec.ch
Device Model Desc               : IEC 61966-2.1 Default RGB colour space - sRGB
Viewing Cond Desc               : Reference Viewing Condition in IEC61966-2.1
Viewing Cond Illuminant         : 19.6445 20.3718 16.8089
Viewing Cond Surround           : 3.92889 4.07439 3.36179
Viewing Cond Illuminant Type    : D50
Luminance                       : 76.03647 80 87.12462
Measurement Observer            : CIE 1931
Measurement Backing             : 0 0 0
Measurement Geometry            : Unknown
Measurement Flare               : 0.999%
Measurement Illuminant          : D65
Technology                      : Cathode Ray Tube Display
Red Tone Reproduction Curve     : (Binary data 2060 bytes, use -b option to extract)
Green Tone Reproduction Curve   : (Binary data 2060 bytes, use -b option to extract)
Blue Tone Reproduction Curve    : (Binary data 2060 bytes, use -b option to extract)
Title                           : Commands Guide
Description                     : Hadoop
Date                            : 2013:07:22 15:25:46-07:00
PDF Version                     : 1.4
Producer                        : Apache FOP Version 0.95
Creator                         : Apache Forrest - http://forrest.apache.org/
Create Date                     : 2013:07:22 15:25:46-07:00
Metadata Date                   : 2013:07:22 15:25:46-07:00
Page Mode                       : UseOutlines
Subject                         : Hadoop

EXIF Metadata provided by EXIF.tools

Commands Guide Manual

commands_manual

Navigation menu

Versions of this User Manual:

Views

Navigation