Commands Manual

commands_manual

User Manual:

Open the PDF directly: View PDF .
Page Count: 10

Download
Open PDF In Browser	View PDF

Commands Guide
Table of contents
1

Overview............................................................................................................................2
1.1

Generic Options.............................................................................................................2

User Commands ................................................................................................................3
2.1

archive .......................................................................................................................... 3

2.2

distcp .............................................................................................................................3

2.3

fs ................................................................................................................................... 3

2.4

fsck ............................................................................................................................... 3

2.5

jar ..................................................................................................................................4

2.6

job ................................................................................................................................. 4

2.7

pipes ..............................................................................................................................5

2.8

queue .............................................................................................................................6

2.9

version .......................................................................................................................... 6

2.10
3

CLASSNAME ............................................................................................................ 6

Administration Commands ............................................................................................... 7
3.1

balancer .........................................................................................................................7

3.2

daemonlog .................................................................................................................... 7

3.3

datanode.........................................................................................................................7

3.4

dfsadmin ....................................................................................................................... 8

3.5

jobtracker ......................................................................................................................9

3.6

namenode ......................................................................................................................9

3.7

secondarynamenode ................................................................................................... 10

3.8

tasktracker ...................................................................................................................10

Commands Guide

1. Overview
All hadoop commands are invoked by the bin/hadoop script. Running the hadoop script
without any arguments prints the description for all commands.
Usage: hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS]
[COMMAND_OPTIONS]
Hadoop has an option parsing framework that employs parsing generic options as well as
running classes.
COMMAND_OPTION

Description

--config confdir

Overwrites the default Configuration directory.
Default is ${HADOOP_HOME}/conf.

GENERIC_OPTIONS

The common set of options supported by
multiple commands.

COMMAND
COMMAND_OPTIONS

Various commands with their options are
described in the following sections. The
commands have been grouped into User
Commands and Administration Commands.

1.1. Generic Options
The following options are supported by dfsadmin, fs, fsck and job. Applications should
implement Tool to support GenericOptions.
GENERIC_OPTION

Description

-conf

Specify an application configuration file.

-D

Use value for given property.

-fs

Specify a namenode.

-jt

Specify a job tracker. Applies only to job.

-files

Specify comma separated files to be copied to
the map reduce cluster. Applies only to job.

-libjars

Specify comma separated jar files to include in
the classpath. Applies only to job.

-archives

Specify comma separated archives to be
unarchived on the compute machines. Applies
only to job.

Page 2

Commands Guide

2. User Commands
Commands useful for users of a hadoop cluster.

2.1. archive
Creates a hadoop archive. More information can be found at Hadoop Archives.
Usage: hadoop archive -archiveName NAME *
COMMAND_OPTION

Description

-archiveName NAME

Name of the archive to be created.

src

Filesystem pathnames which work as usual with
regular expressions.

dest

Destination directory which would contain the
archive.

2.2. distcp
Copy file or directories recursively. More information can be found at Hadoop DistCp Guide.
Usage: hadoop distcp
COMMAND_OPTION

Description

srcurl

Source Url

desturl

Destination Url

2.3. fs
Usage: hadoop fs [GENERIC_OPTIONS] [COMMAND_OPTIONS]
Runs a generic filesystem user client.
The various COMMAND_OPTIONS can be found at Hadoop FS Shell Guide.

2.4. fsck
Runs a HDFS filesystem checking utility. See Fsck for more info.
Usage: hadoop fsck [GENERIC_OPTIONS] [-move | -delete |
-openforwrite] [-files [-blocks [-locations | -racks]]]

Page 3

Commands Guide

COMMAND_OPTION

Description

Start checking from this path.

-move

Move corrupted files to /lost+found

-delete

Delete corrupted files.

-openforwrite

Print out files opened for write.

-files

Print out files being checked.

-blocks

Print out block report.

-locations

Print out locations for every block.

-racks

Print out network topology for data-node
locations.

2.5. jar
Runs a jar file. Users can bundle their Map Reduce code in a jar file and execute it using this
command.
Usage: hadoop jar [mainClass] args...
The streaming jobs are run via this command. Examples can be referred from Streaming
examples
Word count example is also run using jar command. It can be referred from Wordcount
example

Description

-submit

Submits the job.

-status

Prints the map and reduce completion
Page 4

Commands Guide

percentage and all job counters.
-counter

Prints the counter value.

-kill

Kills the job.

-events
<#-of-events>

Prints the events' details received by jobtracker
for the given range.

-history [all]

-history prints job details, failed
and killed tip details. More details about the job
such as successful tasks and task attempts
made for each task can be viewed by specifying
the [all] option.

-list [all]

-list all displays all jobs. -list displays only jobs
which are yet to complete.

-kill-task

Kills the task. Killed tasks are NOT counted
against failed attempts.

-fail-task

Fails the task. Failed tasks are counted against
failed attempts.

-set-priority

Changes the priority of the job. Allowed priority
values are VERY_HIGH, HIGH, NORMAL,
LOW, VERY_LOW

2.7. pipes
Runs a pipes job.
Usage: hadoop pipes [-conf ] [-jobconf ,
, ...] [-input ] [-output ] [-jar ] [-inputformat ] [-map ] [-partitioner
] [-reduce ] [-writer ] [-program
] [-reduces ]
COMMAND_OPTION

Description

-conf

Configuration for job

-jobconf , ,
...

Add/override configuration for job

-input

Input directory

-output

Output directory

Page 5

Commands Guide

-jar

Jar filename

-inputformat

InputFormat class

-map

Java Map class

-partitioner

Java Partitioner

-reduce

Java Reduce class

-writer

Java RecordWriter

-program

Executable URI

-reduces

Number of reduces

2.8. queue
command to interact and view Job Queue information
Usage : hadoop queue [-list] | [-info
[-showJobs]]
COMMAND_OPTION

Description

-list

Gets list of Job Queues configured in the
system. Along with scheduling information
associated with the job queues.

-info [-showJobs]

Displays the job queue information and
associated scheduling information of particular
job queue. If -showJobs options is present a list
of jobs submitted to the particular job queue is
displayed.

2.9. version
Prints the version.
Usage: hadoop version

2.10. CLASSNAME
hadoop script can be used to invoke any class.
Usage: hadoop CLASSNAME
Runs the class named CLASSNAME.
Page 6

Commands Guide

3. Administration Commands
Commands useful for administrators of a hadoop cluster.

3.1. balancer
Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the
rebalancing process. See Rebalancer for more details.
Usage: hadoop balancer [-threshold ]
COMMAND_OPTION
-threshold

Description
Percentage of disk capacity. This overwrites the
default threshold.

3.2. daemonlog
Get/Set the log level for each daemon.
Usage: hadoop daemonlog -getlevel
Usage: hadoop daemonlog -setlevel
COMMAND_OPTION
-getlevel

Description
Prints the log level of the daemon running at
. This command internally connects
to http:///logLevel?log=

-setlevel Sets the log level of the daemon running at
. This command internally connects
to http:///logLevel?log=

3.3. datanode
Runs a HDFS datanode.
Usage: hadoop datanode [-rollback]
COMMAND_OPTION
-rollback

Description
Rollsback the datanode to the previous version.
This should be used after stopping the datanode
and distributing the old hadoop version.

Page 7

Commands Guide

3.4. dfsadmin
Runs a HDFS dfsadmin client.
Usage: hadoop dfsadmin [GENERIC_OPTIONS] [-report] [-safemode
enter | leave | get | wait] [-refreshNodes] [-finalizeUpgrade]
[-upgradeProgress status | details | force] [-metasave
filename] [-setQuota ...] [-clrQuota
...] [-help [cmd]]
COMMAND_OPTION

Description
Reports basic filesystem information and
statistics.

-report

-safemode enter | leave | get | wait Safe mode maintenance command. Safe mode
is a Namenode state in which it
1. does not accept changes to the name space
(read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at
Namenode startup, and leaves safe mode
automatically when the configured minimum
percentage of blocks satisfies the minimum
replication condition. Safe mode can also be
entered manually, but then it can only be turned
off manually as well.
-refreshNodes

Re-read the hosts and exclude files to update
the set of Datanodes that are allowed to connect
to the Namenode and those that should be
decommissioned or recommissioned.

-finalizeUpgrade

Finalize upgrade of HDFS. Datanodes delete
their previous version working directories,
followed by Namenode doing the same. This
completes the upgrade process.

-upgradeProgress status | details |
force

Request current distributed upgrade status, a
detailed status or force the upgrade to proceed.

-metasave filename

Save Namenode's primary data structures to
in the directory specified by
hadoop.log.dir property. will contain
one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated

Page 8

Commands Guide

3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota
...

Set the quota for each directory
. The directory quota is a long integer
that puts a hard limit on the number of names in
the directory tree.
Best effort for the directory, with faults reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
4. the directory would immediately exceed the
new quota.

-clrQuota ...

Clear the quota for each directory .
Best effort for the directory. with fault reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.

-help [cmd]

Displays help for the given command or all
commands if none is specified.

3.5. jobtracker
Runs the MapReduce job Tracker node.
Usage: hadoop jobtracker

3.6. namenode
Runs the namenode. More info about the upgrade, rollback and finalize is at Upgrade
Rollback
Usage: hadoop namenode [-format] | [-upgrade] | [-rollback] |
[-finalize] | [-importCheckpoint]
COMMAND_OPTION

Description

-format

Formats the namenode. It starts the namenode,
formats it and then shut it down.

-upgrade

Namenode should be started with upgrade
option after the distribution of new hadoop
version.

-rollback

Rollsback the namenode to the previous
version. This should be used after stopping the

Page 9

Commands Guide

cluster and distributing the old hadoop version.
-finalize

Finalize will remove the previous state of the
files system. Recent upgrade will become
permanent. Rollback option will not be available
anymore. After finalization it shuts the
namenode down.

-importCheckpoint

Loads image from a checkpoint directory and
save it into the current one. Checkpoint dir is
read from property fs.checkpoint.dir

3.7. secondarynamenode
Runs the HDFS secondary namenode. See Secondary Namenode for more info.
Usage: hadoop secondarynamenode [-checkpoint [force]] |
[-geteditsize]
COMMAND_OPTION

Description

-checkpoint [force]

Checkpoints the Secondary namenode if
EditLog size >= fs.checkpoint.size. If -force is
used, checkpoint irrespective of EditLog size.

-geteditsize

Prints the EditLog size.

3.8. tasktracker
Runs a MapReduce task Tracker node.
Usage: hadoop tasktracker

Page 10

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.3
Linearized                      : No
Page Count                      : 10
Page Mode                       : UseOutlines
Type                            : Info
Producer                        : FOP 0.20.5

EXIF Metadata provided by EXIF.tools

Commands Manual

commands_manual

commands_manual

commands_manual

Navigation menu

Versions of this User Manual:

Views

Navigation