CSCE 215 Unix Ultimate Guide 3E

Your%20UNIX%20Linux%20-%20The%20Ultimate%20Guide%20-%20Third%20Edition

Your%20UNIX%20Linux%20-%20The%20Ultimate%20Guide%20-%20Third%20Edition

Your-UNIX-Linux-The-Ultimate-Guide

Your%20UNIX%20Linux%20-%20The%20Ultimate%20Guide%20-%20Third%20Edition

Your%20UNIX%20Linux%20-%20The%20Ultimate%20Guide%20-%20Third%20Edition

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 801

Download
Open PDF In Browser	View PDF

Third Edition

A separate chapter on the essentials of C programming.
Coverage of the GNU debugger (gdb), which is a more effective, powerful
debugger than dbx, the original UNIX debugger.
The popular Concurrent Version System (CVS), which is found on all
Linux systems.
The bc calculator utility, an efficient computing tool for shell programmers.
Coverage of the pico editor and relocates the chapter on emacs to an appendix.
The popular Red Hat Package manager (RPM).
A wealth of real-life examples.
Numerous learning aids such as Notes, Tips, Caution boxes, and a separate
icon that highlights Linux references.

Your
UNIX/Linux
The Ultimate Guide

Das

MD DALIM #1173349 12/18/11 CYAN MAG YELO BLK

Visit the Your UNIX/Linux website at www.mhhe.com/das for solutions, slides,
source code, links, and more.

Your UNIX/Linux

The new edition of Your UNIX/Linux includes:

The Ultimate Guide

Your UNIX/Linux: The Ultimate Guide, Third Edition, is written with both users
and programmers in mind and is the ultimate UNIX/Linux text. Both pedagogical
tool and exhaustive reference, it is well suited to any course that includes UNIX
or Linux. A strong pedagogical framework sets it apart from similar texts and
allows beginning students to gain a firm grasp of fundamental concepts, while
chapters on advanced topics inspire the more experienced reader to move
beyond the basics. Nearly a thousand exercises and self-test questions provide
a way for students to test and reinforce their understanding of the material.

Third Edition

Sumitabha Das

Your UNIX/
Linux
The Ultimate Guide
Third Edition

Sumitabha Das

das76205_fm_i-xI.indd i

12/14/11 4:36 PM

YOUR UNIX/LINUX: THE ULTIMATE GUIDE, THIRD EDITION
Published by McGraw-Hill, a business unit of The McGraw-Hill Companies, Inc., 1221 Avenue of the Americas,
New York, NY 10020. Copyright © 2013 by the McGraw-Hill Companies, Inc. All rights reserved. Previous
editions © 2006 and 2001. Printed in the United States of America. No part of this publication may be reproduced
or distributed in any form or by any means, or stored in a database or retrieval system, without the prior
written consent of The McGraw-Hill Companies, Inc., including, but not limited to, in any network or other
electronic storage or transmission, or broadcast for distance learning.
Some ancillaries, including electronic and print components, may not be available to customers outside the
United States.
This book is printed on acid-free paper.
1234567890

DOC/DOC

1098765432

ISBN 978–0–07–337620–2
MHID
0–07–337620–5
Vice President & Editor-in-Chief: Marty Lange
Vice President of Specialized Production: Janice M. Roerig-Blong
Editorial Director: Michael Lange
Publisher: Raghothaman Srinivasan
Marketing Manager: Curt Reynolds
Project Manager: Melissa M. Leick
Design Coordinator: Brenda A. Rolwes
Cover Design: Studio Montage, St. Louis, Missouri
Cover Image: © Creatas/PunchStock RF
Buyer: Sherry L. Kane
Media Project Manager: Balaji Sundararaman
Compositor: Cenveo Publisher Services
Typeface: 10/12 Times Roman
Printer: R. R. Donnelley
All credits appearing on page or at the end of the book are considered to be an extension of the copyright page.
Library of Congress Cataloging-in-Publication Data
Das, Sumitabha.
Your UNIX/LINUX : the ultimate guide / Sumitabha Das. — 3rd ed.
p. cm.
Rev. ed. of: Your UNIX : the ultimate guide / Sumitabha Das. 2006.
ISBN-13: 978-0-07-337620-2 (alk. paper)
ISBN-10: 0-07-337620-5 (alk. paper)
1. UNIX (Computer file) 2. Linux. 3. Operating systems (Computers) I.
Das, Sumitabha. Your UNIX. II. Title.
QA76.76.O63D3495
2013
005.4’32—dc23
2011042979

www.mhhe.com

das76205_fm_i-xI.indd ii

12/14/11 4:36 PM

To my wife Julie and daughter Sohini, whose
contribution can no longer be ignored.

das76205_fm_i-xI.indd iii

12/14/11 4:36 PM

iv

das76205_fm_i-xI.indd iv

12/14/11 4:36 PM

Contents in Brief

PART I
1
2
3
4
5
6
7
8
9
10
11

UNIX for the User ...........................................................................................1
Introducing UNIX ...........................................................................................3
Becoming Familiar with UNIX Commands ....................................................25
The File System.............................................................................................56
File Attributes ...............................................................................................92
The vi/vim Editor .......................................................................................122
The Shell ....................................................................................................156
The Process ................................................................................................188
The Shell—Customizing the Environment ...................................................213
Simple Filters ..............................................................................................241
Filters Using Regular Expressions—grep and sed .........................................265
Networking Tools .......................................................................................296

PART II
12
13
14
15
16
17
18

UNIX for the Programmer...........................................................................329
Filtering and Programming with awk ...........................................................331
Shell Programming .....................................................................................359
perl—The Master Manipulator ...................................................................404
Introducing C .............................................................................................440
Program Development Tools.......................................................................485
Systems Programming I—Files ....................................................................535
Systems Programming II—Process Control..................................................570

PART III
19

System Administration ................................................................................607
System Administration ................................................................................609

v

das76205_fm_i-xI.indd v

12/14/11 4:36 PM

Your UNIX/Linux: The Ultimate Guide

vi

A
B
C
D
E
F
G
H
I

das76205_fm_i-xI.indd vi

Appendixes
The C Shell—Programming Constructs .......................................................643
The Korn and Bash Shells—Exclusive Programming Constructs ...................650
The GNU emacs Editor ................................................................................657
vi/vim Command Reference .......................................................................686
The Regular Expression Superset .................................................................693
The HOWTO...............................................................................................696
The ASCII Character Set ..............................................................................701
Glossary......................................................................................................705
Solutions to Self-Test Questions ..................................................................728

12/14/11 4:36 PM

Contents

List of Tables .................................................................................. xxix
Preface ..........................................................................................xxxiii

PART I

UNIX for the User .......................................................................................1

Chapter 1

Introducing UNIX .......................................................................................3
1.1 The Operating System ..............................................................................3
1.2 The UNIX Operating System .....................................................................5
1.3 Knowing Your Machine ............................................................................6
1.4 The System Administrator .........................................................................7
1.5 Logging In and Out ..................................................................................7
1.5.1 Logging In......................................................................................7
1.5.2 Logging Out ...................................................................................8
1.6 A Hands-On Session..................................................................................9
1.6.1 System Information with date and who ..........................................10
1.6.2 Viewing Processes with ps .............................................................10
1.6.3 Handling Files ...............................................................................11
1.6.4 Handling Directories .....................................................................12
1.7 How It All Clicked ...................................................................................14
1.7.1 Berkeley: The Second School ..........................................................15
1.7.2 UNIX Gets Fragmented .................................................................15
1.7.3 The Internet .................................................................................16
1.7.4 The Windows Threat .....................................................................16
1.8 POSIX and the Single UNIX Specification ................................................16
1.9 Linux and GNU.......................................................................................17
1.10 The UNIX Architecture ..........................................................................17
1.10.1 Division of Labor: Kernel and Shell ...............................................18

vii

das76205_fm_i-xI.indd vii

12/14/11 4:36 PM

viii

Your UNIX/Linux: The Ultimate Guide
1.10.2 The File and Process ....................................................................19
1.10.3 The System Calls.........................................................................19
1.11 Features of UNIX ..................................................................................20
1.11.1 A Multiuser System .....................................................................20
1.11.2 A Multitasking System Too ..........................................................20
1.11.3 A Repository of Applications .........................................................21
1.11.4 The Building-Block Approach .......................................................21
1.11.5 Pattern Matching .......................................................................21
1.11.6 Programming Facility ..................................................................21
1.11.7 Documentation ..........................................................................22
Summary ......................................................................................................22
Self-Test ........................................................................................................23
Exercises .......................................................................................................24

Chapter 2

Becoming Familiar with UNIX Commands ...........................................25
2.1 Command Basics ....................................................................................26
2.1.1 The PATH: Locating Commands ......................................................26
2.1.2 Where Is the Command? ...............................................................27
2.2 Command Structure ...............................................................................28
2.3 Flexibility of Command Usage ................................................................30
2.4 man: On-Line Help ...................................................................................31
2.4.1 Navigation and Search .................................................................33
2.4.2 Further Help with man -k and man -f ............................................33
2.5 The man Documentation .........................................................................34
2.5.1 Understanding a man Page ...........................................................35
2.5.2 Using man to Understand man ........................................................35
2.6 echo: Displaying Messages ......................................................................38
2.7 printf: Alternative to echo .....................................................................39
2.8 script: Recording Your Session ..............................................................40
2.9 Using Email with mailx ...........................................................................40
2.9.1 Sending Mail ................................................................................41
2.9.2 Receiving Mail ..............................................................................41
2.9.3 mailx Internal Commands ............................................................42
2.10 passwd: Changing Your Password..........................................................44
2.11 uname: Your Machine’s Name and Operating System ............................44
2.12 who: Know the Users .............................................................................45

das76205_fm_i-xI.indd viii

12/14/11 4:36 PM

Contents

ix
2.13 date: Displaying the System Date .........................................................46
2.14 stty: When Things Go Wrong ..............................................................47
2.14.1 Changing the Settings.................................................................48
2.15 The X Window System ..........................................................................49
2.15.1 The Terminal Emulator ................................................................50
2.15.2 The File Manager ........................................................................51
Summary ......................................................................................................52
Self-Test ........................................................................................................53
Exercises .......................................................................................................54

Chapter 3

The File System .........................................................................................56
3.1 The File...................................................................................................57
3.1.1 Ordinary (Regular) File ..................................................................57
3.1.2 Directory File ................................................................................58
3.1.3 Device File ....................................................................................58
3.2 What’s in a (File)name? ...........................................................................58
3.3 The File System Hierarchy ......................................................................59
3.4 The UNIX File System .............................................................................60
3.5 Using Absolute Pathnames with Commands ...........................................61
3.6 The HOME Variable and ~: The Home Directory .........................................62
3.7 pwd and cd: Navigating the File System ...................................................62
3.8 Relative Pathnames (. and ..) ................................................................64
3.9 mkdir: Making Directories .......................................................................66
3.10 rmdir: Removing Directories .................................................................67
3.11 ls: Listing Files......................................................................................68
3.11.1 ls Options .................................................................................69
3.12 cp: Copying Files ..................................................................................71
3.12.1 cp Options .................................................................................72
3.13 mv: Renaming Files ................................................................................73
3.14 rm: Deleting Files ..................................................................................73
3.14.1 rm Options .................................................................................74
3.15 cat: Displaying and Concatenating Files ...............................................76
3.16 more: The UNIX Pager ...........................................................................76
3.17 pico: Rudimentary Text Editing.............................................................78
3.17.1 Navigation .................................................................................79
3.17.2 Text Editing ................................................................................80

das76205_fm_i-xI.indd ix

12/14/11 4:36 PM

Your UNIX/Linux: The Ultimate Guide

x

3.18 wc: Counting Lines, Words and Characters ............................................80
3.19 lp: Printing a File ..................................................................................81
3.19.1 Other Commands in the lp Subsystem..........................................81
3.20 od: Viewing Nonprintable Characters ....................................................82
3.21 dos2unix, unix2dos, and Tofrodos:
Converting Between DOS and UNIX .................................................83
3.22 tar: The Archival Program ....................................................................84
3.23 gzip: The Compression Program ..........................................................85
3.24 zip: The Compression and Archival Program ........................................87
3.25 Other Ways of Using These Commands ................................................88
Summary ......................................................................................................88
Self-Test ........................................................................................................89
Exercises .......................................................................................................90
Chapter 4

das76205_fm_i-xI.indd x

File Attributes ...........................................................................................92
4.1 ls Revisited (-l): Listing File Attributes ...................................................92
4.1.1 Listing Directory Attributes (-ld) ....................................................94
4.2 File Permissions.......................................................................................94
4.3 chmod: Changing File Permissions ............................................................96
4.3.1 Relative Permissions ......................................................................96
4.3.2 Absolute Assignment .....................................................................98
4.3.3 Recursive Operation (-R) ...............................................................99
4.4 The Directory .......................................................................................100
4.4.1 Read Permission .........................................................................100
4.4.2 Write Permission .........................................................................100
4.4.3 Execute Permission ......................................................................102
4.5 umask: Default File and Directory Permissions ........................................102
4.6 File Systems and Inodes ........................................................................103
4.7 ln: Creating Hard Links .........................................................................105
4.7.1 Where to Use Hard Links .............................................................107
4.8 ln Again: Creating Symbolic Links ........................................................107
4.9 File Ownership......................................................................................109
4.9.1 chown: Changing File Ownership ..................................................110
4.9.2 chgrp: Changing Group Owner...................................................111
4.9.3 How to Handle Intruders .............................................................112
4.10 Modification and Access Times ...........................................................112

12/14/11 4:36 PM

Contents

xi
4.11 find: Locating Files...............................................................................113
4.11.1 Selection Criteria.......................................................................114
4.11.2 The find Operators (!, -o, and -a) .............................................116
4.11.3 Operators of the Action Component ...........................................117
Summary ....................................................................................................118
Self-Test ......................................................................................................118
Exercises .....................................................................................................119

Chapter 5

The vi/vim Editor....................................................................................122
5.1 vi Basics ...............................................................................................123
5.1.1 The File .exrc ............................................................................124
5.2 A Few Tips First .....................................................................................125
5.3 Input Mode—Entering and Replacing Text ...........................................126
5.3.1 Inserting and Appending Text (i and a) .......................................127
5.3.2 Inserting and Appending Text at Line Extremes (I and A) ..............127
5.3.3 Opening a New Line (o and O) ....................................................128
5.3.4 Replacing Text (r, s, R, and S) ....................................................129
5.3.5 Entering Control Characters ([Ctrl-v]) ...........................................130
5.4 Saving Text and Quitting—The ex Mode ..............................................131
5.4.1 Saving Your Work (:w) ................................................................132
5.4.2 Saving and Quitting (:x and :wq)................................................132
5.4.3 Aborting Editing (:q) ...................................................................132
5.4.4 Writing Selected Lines .................................................................133
5.4.5 Escape to the Shell (:sh and [Ctrl-z]) ...........................................134
5.5 Recovering from a Crash (:recover and -r) ..........................................134
5.6 Navigation............................................................................................134
5.6.1 Relative Movement in the Four Directions (h, j, k, and l) .............135
5.6.2 Scrolling ([Ctrl-f], [Ctrl-b], [Ctrl-d] and [Ctrl-u]) ............................136
5.6.3 Word Navigation (b, e, and w) ....................................................136
5.6.4 Moving to Line Extremes (0, |, and $)..........................................137
5.6.5 Absolute Movement (G) ...............................................................137
5.7 Editing Text without Operators .............................................................138
5.7.1 Deleting Text (x, X, and dd) ........................................................138
5.7.2 Moving Text (p) ..........................................................................139
5.7.3 Joining Lines (J) ..........................................................................139
5.7.4 Changing Case (~) ......................................................................140

das76205_fm_i-xI.indd xi

12/14/11 4:36 PM

xii

Your UNIX/Linux: The Ultimate Guide
5.8 Correcting a C Program ........................................................................140
5.9 Editing Text with Operators ..................................................................141
5.9.1 Deleting and Moving Text (d, p, and P)........................................142
5.9.2 Yanking Text (y, p and P) ............................................................143
5.9.3 Changing Text (c) .......................................................................144
5.10 Copying and Moving Text from One File to Another...........................144
5.11 Undoing Last Editing Instructions (u and U).........................................145
5.12 Searching for a Pattern (/ and ?).........................................................146
5.12.1 Repeating the Last Pattern Search (n and N) ...............................146
5.13 Repeating the Last Command (.) .......................................................147
5.14 Substitution—Search and Replace (:s) ................................................148
5.15 set: Customizing vi ...........................................................................149
Going Further .....................................................................................151
5.16 map: Mapping Keys of Keyboard..........................................................151
5.17 abbr: Abbreviating Text Input .............................................................152
Summary ....................................................................................................152
Self-Test ......................................................................................................153
Exercises .....................................................................................................154

Chapter 6

das76205_fm_i-xI.indd xii

The Shell ..................................................................................................156
6.1 The Shell as Command Processor .........................................................157
6.2 Shell Offerings ......................................................................................158
6.3 Pattern Matching—The Wild Cards .......................................................158
6.3.1 The * and ? ................................................................................158
6.3.2 The Character Class ....................................................................160
6.3.3 Matching the Dot .......................................................................162
6.3.4 Rounding Up ..............................................................................162
6.4 Escaping and Quoting ..........................................................................163
6.4.1 Escaping ....................................................................................163
6.4.2 Quoting .....................................................................................164
6.4.3 Escaping in echo .........................................................................165
6.5 Redirection ...........................................................................................165
6.5.1 Standard Input ...........................................................................166
6.5.2 Standard Output ........................................................................168
6.5.3 The File Descriptor ......................................................................169
6.5.4 Standard Error ............................................................................170
6.5.5 Filters—Using Both Standard Input and Standard Output..............171

12/14/11 4:36 PM

Contents

xiii
6.6 Collective Manipulation ........................................................................172
6.6.1 Replicating Descriptors ................................................................172
6.6.2 Command Grouping ...................................................................172
6.7 /dev/null and /dev/tty: Two Special Files ...........................................173
6.8 Pipes.....................................................................................................174
6.8.1 When a Command Needs to Be Ignorant of Its Source...................175
6.9 tee: Creating a Tee ...............................................................................176
6.10 Command Substitution ......................................................................177
6.11 Shell Variables .....................................................................................178
6.11.1 Effects of Quoting and Escaping .................................................180
6.11.2 Where to Use Shell Variables......................................................180
6.12 Shell Scripts ........................................................................................181
6.13 The Shell’s Treatment of the Command Line .......................................182
Going Further .....................................................................................183
6.14 More Wild Cards.................................................................................183
6.15 xargs: Building a Dynamic Command Line .........................................183
Summary ....................................................................................................184
Self-Test ......................................................................................................185
Exercises .....................................................................................................186

Chapter 7

das76205_fm_i-xI.indd xiii

The Process ..............................................................................................188
7.1 Process Basics .......................................................................................188
7.2 The Shell and init................................................................................189
7.3 ps: Displaying Process Attributes ...........................................................190
7.5 The Process Creation Mechanism..........................................................194
7.6 Inherited Process Attributes ..................................................................195
7.6.1 When Variables Are Inherited and When They Are Not ..................196
7.7 When You Can’t Use a Separate Process................................................197
7.8 Process States and Zombies ..................................................................198
7.8.1 ps -l: Detailed Process Listing ....................................................198
7.9 Signal Handling ....................................................................................199
7.10 Running Jobs in Background ...............................................................201
7.10.1 &: No Logging Out ....................................................................201
7.10.2 nohup: Log Out Safely ...............................................................202
7.11 Job Control .........................................................................................202
7.11.1 Handling Standard Input and Standard Output ..........................205
7.12 at and batch: Execute Later ................................................................206
7.12.1 at: One-Time Execution.............................................................206

12/14/11 4:36 PM

xiv

Your UNIX/Linux: The Ultimate Guide
7.12.2 batch: Execute in Batch Queue ..................................................206
7.12.3 Restricting Use of at and batch .................................................207
7.13 cron and crontab: Running Jobs Periodically .......................................207
7.13.1 Controlling Access to cron .........................................................210
Summary ....................................................................................................210
Self-Test ......................................................................................................210
Exercises .....................................................................................................211

Chapter 8

das76205_fm_i-xI.indd xiv

The Shell—Customizing the Environment ..........................................213
8.1 The Shells .............................................................................................213
8.1.1 Setting Your Shell........................................................................214
8.2 Environment Variables ..........................................................................215
8.2.1 export: Creating Environment Variables .......................................216
8.3 The Common Environment Variables ....................................................216
8.4 Aliases ..................................................................................................220
8.5 Command History ................................................................................222
8.5.1 Accessing Previous Commands.....................................................223
8.5.2 Substitution in Previous Commands .............................................223
8.5.3 Using Arguments to Previous Command ($_) ................................224
8.5.4 The History Variables ..................................................................224
8.6 In-line Command Editing......................................................................225
8.7 Tilde Substitution..................................................................................226
8.8 Using set Options ................................................................................227
8.9 The Initialization Scripts ........................................................................229
8.9.1 The Login Script ..........................................................................229
8.9.2 The rc File...................................................................................230
8.10 The C Shell .........................................................................................231
8.10.1 Local and Environment Variables ...............................................231
8.10.2 Aliases .....................................................................................233
8.10.3 History .....................................................................................234
8.10.4 Other Shell Variables .................................................................234
8.10.5 The Initialization Scripts ............................................................235
Going Further .....................................................................................235
8.11 Directory Stack Manipulation..............................................................235
Summary ....................................................................................................237
Self-Test ......................................................................................................238
Exercises .....................................................................................................239

12/14/11 4:36 PM

Contents
Chapter 9

xv
Simple Filters...........................................................................................241
9.1 pr: Paginating Files ...............................................................................242
9.1.1 pr Options .................................................................................242
9.2 Comparing Files....................................................................................243
9.3 cmp: Byte-by-Byte Comparison ..............................................................244
9.4 comm: What Is Common? .......................................................................245
9.5 diff: Converting One File to Another ...................................................246
9.6 head: Displaying the Beginning of a File ................................................247
9.7 tail: Displaying the End of a File ..........................................................248
9.7.1 tail Options ..............................................................................248
9.8 cut: Slitting a File Vertically ...................................................................249
9.9 paste: Pasting Files ...............................................................................250
9.10 sort: Ordering a File ...........................................................................251
9.10.1 sort Options ............................................................................252
9.11 uniq: Locate Repeated and Nonrepeated Lines ...................................255
9.11.1 uniq Options ............................................................................255
9.12 tr: Translating Characters ...................................................................257
9.12.1 tr Options ...............................................................................258
9.13 Applying the Filters .............................................................................259
9.13.1 Listing the Five Largest Files in the Current Directory ...................259
9.13.2 Creating a Word-Usage List .......................................................260
9.13.3 Finding Out the Difference between Two Password Files...............261
Summary ....................................................................................................262
Self-Test ......................................................................................................263
Exercises .....................................................................................................263

Chapter 10

das76205_fm_i-xI.indd xv

Filters Using Regular Expressions—grep and sed ................................265
10.1 The Sample Database .........................................................................266
10.2 grep: Searching for a Pattern ..............................................................266
10.2.1 Quoting in grep........................................................................267
10.2.2 When grep Fails .......................................................................268
10.3 grep Options ......................................................................................268
10.4 Basic Regular Expressions (BRE)—An Introduction ...............................271
10.4.1 The Character Class ..................................................................271
10.4.2 The * .......................................................................................273
10.4.3 The Dot ...................................................................................274

12/14/11 4:36 PM

xvi

Your UNIX/Linux: The Ultimate Guide
10.4.4 Specifying Pattern Locations (^ and $)........................................274
10.4.5 When Metacharacters Lose Their Meaning .................................. 275
10.5 Extended Regular Expressions (ERE) and egrep ...................................276
10.5.1 The + and ? ..............................................................................276
10.5.2 Matching Multiple Patterns (|, ( and )) .....................................277
10.6 sed:The Stream Editor.........................................................................277
10.7 Line Addressing ..................................................................................279
10.8 sed Options ........................................................................................280
10.9 Context Addressing ............................................................................281
10.10 Writing Selected Lines to a File (w) ....................................................282
10.11 Text Editing ......................................................................................283
10.11.1 Inserting and Changing Lines (i, a, c) ......................................283
10.11.2 Deleting Lines (d)....................................................................284
10.12 Substitution (s) .................................................................................284
10.12.1 Using Regular Expressions in Substitution .................................285
10.12.2 The Remembered Pattern (//) .................................................286
10.13 Basic Regular Expressions Revisited ...................................................287
10.13.1 The Repeated Pattern (&) ........................................................287
10.13.2 Interval Regular Expression (IRE) ..............................................288
10.13.3 The Tagged Regular Expression (TRE) .......................................288
10.14 Applying the IRE and TRE..................................................................289
10.14.1 Handling a Telephone Directory ...............................................289
10.14.2 Replacing an Obsolescent Function
with a POSIX-Compliant One...................................................290
10.14.3 Converting Pathnames in URLs ................................................291
Summary ....................................................................................................292
Self-Test ......................................................................................................292
Exercises .....................................................................................................293

Chapter 11

das76205_fm_i-xI.indd xvi

Networking Tools ...................................................................................296
11.1 TCP/IP Basics ......................................................................................297
11.1.1 Hostnames and IP Addresses .....................................................297
11.2 Resolving Hostnames and IP Addresses ...............................................298
11.2.1 /etc/hosts: The Hosts File ........................................................298
11.2.2 The Domain Name System (DNS) ..............................................299

12/14/11 4:36 PM

Contents

xvii
11.3 Client-Server: How Networked Applications Communicate .................301
11.4 ping: Checking the Network ...............................................................303
11.5 telnet: Remote Login.........................................................................303
11.6 ftp: File Transfer Protocol ...................................................................304
11.6.1 Basic File and Directory Handling ...............................................305
11.6.2 Transferring Files .......................................................................305
11.6.3 Anonymous FTP ........................................................................306
11.7 Cryptography Basics ...........................................................................307
11.7.1 Symmetric Key Algorithms .........................................................308
11.7.2 Asymmetric Key Algorithms .......................................................308
11.8 SSH: The Secure Shell .........................................................................309
11.8.1 Host Authentication ..................................................................309
11.8.2 The rhosts/shosts Authentication Scheme ................................310
11.8.3 User Authentication with Symmetric Algorithms ..........................310
11.8.4 Using the SSH Agent for Noninteractive Logins ...........................312
11.9 The SSH Tools .....................................................................................313
11.9.1 Remote Login and Command Execution (ssh and slogin) ...........314
11.9.2 File Transfer with sftp and scp ..................................................314
11.10 Internet Mail .....................................................................................315
11.10.1 ~/.signature and ~/.forward: Two Important Files ................317
11.11 MIME: Handling Binary Attachments in Mail .....................................317
11.12 Using X Window on a TCP/IP Network .............................................318
11.12.1 The Display ............................................................................319
11.12.2 Using the DISPLAY variable .....................................................319
11.12.3 Using the -display Option .....................................................320
11.13 HTTP and the World Wide Web ........................................................320
11.13.1 The Uniform Resource Locator (URL) ........................................321
11.13.2 HTTP: The Protocol of the Web ................................................322
11.13.3 Running External Programs .....................................................322
11.13.4 HTML: The Language of Web Pages .........................................323
11.13.5 The Web Browser ....................................................................323
11.14 Multimedia on the Web: MIME Revisited ..........................................324
Summary ....................................................................................................325
Self-Test ......................................................................................................326
Exercises .....................................................................................................327

das76205_fm_i-xI.indd xvii

12/14/11 4:36 PM

xviii

PART II
Chapter 12

Your UNIX/Linux: The Ultimate Guide
UNIX for the Programmer.....................................................................329
Filtering and Programming with awk ..................................................331
12.1 awk Preliminaries .................................................................................332
12.2 Using print and printf ......................................................................334
12.2.1 Redirecting Standard Output .....................................................335
12.3 Number Processing ............................................................................335
12.4 Variables and Expressions....................................................................337
12.5 The Comparison and Logical Operators ..............................................338
12.5.1 String and Numeric Comparison ................................................338
12.5.2 ~ and !~: The Regular Expression Operators ...............................340
12.5.3 The Logical Operators ...............................................................340
12.6 The -f Option: Storing awk Programs in a File.....................................342
12.7 The BEGIN and END Sections ................................................................342
12.8 Positional Parameters ..........................................................................343
12.9 Arrays .................................................................................................344
12.9.1 Associative (Hash) Arrays ..........................................................345
12.9.2 ENVIRON[ ]: The Environment Array ..........................................345
12.10 Built-In Variables ...............................................................................346
12.10.1 Applying the Built-in Variables .................................................347
12.11 Functions ..........................................................................................348
12.12 Control Flow—The if Statement ......................................................350
12.13 Looping with for ..............................................................................351
12.13.1 Using for with an Associative Array .........................................352
12.14 Looping with while ..........................................................................353
12.15 Conclusion .......................................................................................354
Summary ....................................................................................................355
Self-Test ......................................................................................................355
Exercises .....................................................................................................356

Chapter 13

das76205_fm_i-xI.indd xviii

Shell Programming ................................................................................359
13.1 Shell Scripts ........................................................................................360
13.1.1 script.sh: A Simple Script .......................................................360
13.1.2 The She-Bang Line ....................................................................361
13.2 read: Making Scripts Interactive ..........................................................361
13.3 Using Command-Line Arguments .......................................................362
13.4 exit and $?: Exit Status of a Command ..............................................364

12/14/11 4:36 PM

Contents

xix
13.5 The Logical Operators && and ||—Conditional Execution ...................365
13.6 The if Conditional .............................................................................366
13.7 Using test and [ ] to Evaluate Expressions ........................................367
13.7.1 Numeric Comparison ................................................................367
13.7.2 [ ]: Shorthand for test ...........................................................369
13.7.3 String Comparison ....................................................................369
13.7.4 File Attribute Tests .....................................................................370
13.7.5 Using Compound Conditions .....................................................371
13.8 Two Important Programming Idioms ..................................................372
13.8.1 Running a Task Both Interactively and Noninteractively ...............372
13.8.2 Calling a Script by Different Names ($0) ....................................373
13.9 The case Conditional ..........................................................................374
13.9.1 Using Wild Cards and the | .......................................................375
13.10 expr, bc and basename: Computation and String Handling................376
13.10.1 Computation with expr...........................................................376
13.10.2 Floating-Point Computation with bc .........................................377
13.10.3 String Handling with expr .......................................................378
13.10.4 basename: Changing Filename Extensions .................................379
13.11 Sample Program 1: Automatically Selects Last C Program .................379
13.12 for: Looping with a List ....................................................................380
13.12.1 Sources of the List ...................................................................381
13.12.2 Important Applications Using for ............................................382
13.13 while: Looping .................................................................................382
13.13.1 Handling Redirection in a Loop ................................................383
13.13.2 Using while to Wait for a File ..................................................384
13.13.3 Finding Out Users’ Space Consumption ....................................385
13.13.4 break and continue ...............................................................386
13.14 Sample Script 2: Providing Numeric Extensions to Backup Files ........387
13.15 Manipulating Positional Parameters with set and shift....................389
13.15.1 Killing a Process by Name........................................................390
13.15.2 shift: Shifting Positional Parameters Left.................................391
13.15.3 The IFS Variable: set’s Default Delimiter .................................392
13.16 Sample Script 3: A Table of Contents for Downloaded Files...............392
Going Further .....................................................................................394
13.17 The Here Document (<<) ..................................................................394
13.18 Shell Functions .................................................................................395
13.19 trap: How a Script Handles Signals ...................................................397

das76205_fm_i-xI.indd xix

12/14/11 4:36 PM

xx

Your UNIX/Linux: The Ultimate Guide
13.20 eval: Evaluating Twice ......................................................................398
13.21 The exec Statement ..........................................................................399
Summary ....................................................................................................399
Self-Test ......................................................................................................400
Exercises .....................................................................................................401

Chapter 14

das76205_fm_i-xI.indd xx

perl—The Master Manipulator .............................................................404
14.1 perl Preliminaries ...............................................................................405
14.2 Variables and Constants ......................................................................406
14.2.1 Using a Pragma........................................................................407
14.3 Operators ...........................................................................................408
14.3.1 The Comparison and Logical Operators ......................................408
14.3.2 The Concatenation and Repetition Operators (. and x) ...............409
14.4 The Standard Conditionals and Loops.................................................410
14.4.1 The if Conditional ...................................................................410
14.4.2 The while, do-while, and for Loops..........................................410
14.5 Reading Files from Command-Line Arguments ...................................412
14.5.1 Reading Files with One-Liners ....................................................412
14.5.2 Reading Files in a Script .............................................................413
14.6 The Current Line Number ($.) and the Range Operator (..) ..............414
14.7 $_: The Default Variable ......................................................................414
14.8 String-Handling Functions ..................................................................415
14.9 Lists and Arrays ...................................................................................416
14.9.1 Array-Handling Functions ..........................................................419
14.10 foreach: Looping Through a List ......................................................420
14.11 Two Important List Functions............................................................421
14.11.1 split: Splitting into a List or Array ..........................................421
14.11.2 join: Joining a List..................................................................422
14.12 dec2bin.pl: Converting a Decimal Number to Binary .......................423
14.13 Associative Arrays .............................................................................424
14.13.1 Counting Number of Occurrences ............................................425
14.14 Using Regular Expressions .................................................................426
14.14.1 Identifying Whitespace, Digits, and Words ................................426
14.14.2 The IRE and TRE Features ........................................................427
14.15 Substitution with the s and tr Functions ..........................................428
14.15.1 Editing Files In-Place................................................................429
14.16 File Handling ....................................................................................430

12/14/11 4:36 PM

Contents

xxi
14.17 File Tests ...........................................................................................431
14.18 Subroutines ......................................................................................432
Going Further .....................................................................................434
14.19 CGI Programming with perl—An Overview .....................................434
14.19.1 The Query String ....................................................................434
14.19.2 GET and POST: The Request Method ........................................434
Summary ....................................................................................................435
Self-Test ......................................................................................................436
Exercises .....................................................................................................437

Chapter 15

Introducing C ..........................................................................................440
15.1 The C Language .................................................................................440
15.2 first_prog.c: Understanding Our First C Program...............................441
15.2.1 The Preprocessor Section ...........................................................443
15.2.2 The Body ..................................................................................443
15.2.3 main: A Unique Function ...........................................................444
15.3 printf and scanf: Two Important Functions.......................................445
15.3.1 printf: Printing to the Terminal ................................................445
15.3.2 scanf: Input from the Keyboard.................................................446
15.4 Variables and Constants ......................................................................446
15.5 The Fundamental Data Types..............................................................447
15.5.1 The Integer Types .....................................................................448
15.5.2 The Floating-Point Types ...........................................................449
15.5.3 The Character Type...................................................................449
15.6 sizeof.c: Determining the Size of the Fundamental Data Types .........450
15.7 Symbolic Constants: Constants with Names .......................................451
15.8 Arrays .................................................................................................452
15.9 arrays.c: Printing Array Elements.......................................................452
15.10 Structures .........................................................................................454
15.11 Operators and Expressions ................................................................455
15.11.1 The Bitwise Operators .............................................................456
15.11.2 Automatic Conversion .............................................................456
15.11.3 Explicit Conversion ..................................................................457
15.12 type_conversion.c: Demonstrates Type Conversion .........................458
15.13 Order of Evaluation...........................................................................460
15.13.1 Operator Precedence ...............................................................460
15.13.2 Operator Associativity .............................................................461

das76205_fm_i-xI.indd xxi

12/14/11 4:36 PM

xxii

Your UNIX/Linux: The Ultimate Guide
15.14 Control Flow Revisited ......................................................................462
15.14.1 Decision Making with switch ..................................................462
15.14.2 Repetition with do-while .........................................................463
15.14.3 break and continue ...............................................................464
15.15 Functions ..........................................................................................465
15.16 first_func.c: Two Arguments and Return Value ................................466
15.17 Function Arguments, Parameters, and Local Variables .......................467
15.17.1 Passing by Value .....................................................................468
15.17.2 swap_failure.c: Visibility of Parameters and Local Variables ....469
15.17.3 Using Arrays as Function Arguments ........................................470
15.17.4 Using Structures as Function Arguments ...................................471
15.18 Pointers ............................................................................................472
15.19 pointers.c: Basic Features of a Pointer .............................................474
15.20 Pointer Attributes..............................................................................475
15.20.1 Pointer Arithmetic ...................................................................475
15.20.2 Other Attributes......................................................................476
15.21 Pointers and Functions......................................................................477
15.21.1 Solving the Swapping Problem.................................................477
15.21.2 Using Pointers to Return Multiple Values...................................478
15.22 Pointers and Other Data Types .........................................................478
15.22.1 Pointers and Arrays .................................................................478
15.22.2 Pointers and Strings ................................................................479
15.22.3 Array of Strings .......................................................................480
15.22.4 Pointers and Structures ...........................................................480
Summary ....................................................................................................481
Self-Test ......................................................................................................482
Exercises .....................................................................................................483

Chapter 16

das76205_fm_i-xI.indd xxii

Program Development Tools ................................................................485
16.1 Handling Multisource C Applications ..................................................485
16.1.1 A Multisource Application ..........................................................486
16.1.2 Compiling and Linking the Application .......................................489
16.1.3 Why We Need the .o Files .........................................................490
16.2 make: Keeping Programs Up to Date ...................................................490
16.2.1 Removing Redundancies ............................................................493
16.2.2 Other Functions of make: Cleaning Up and Backup ......................493
16.2.3 Macros ....................................................................................494

12/14/11 4:36 PM

Contents

xxiii
16.3 ar: Building a Library (Archive) ...........................................................495
16.3.1 Using the Library ......................................................................496
16.3.2 Maintaining an Archive with make..............................................496
16.4 Static and Shared Libraries ..................................................................497
16.5 Version Control with SCCS, RCS, and CVS ..........................................498
16.6 An SCCS Session .................................................................................499
16.6.1 admin: Creating an SCCS File .....................................................500
16.6.2 get: Checking Out ....................................................................501
16.6.3 delta: Checking In ...................................................................502
16.6.4 unget: When You Change Your Mind .........................................504
16.7 Activity and History Information .........................................................504
16.7.1 sact: Displaying Activity Status .................................................504
16.7.2 prs: Displaying the SCCS History ...............................................505
16.8 Continuing Editing with SCCS ............................................................506
16.8.1 get -r: Changing the Default Numbering Sequence ...................506
16.8.2 Working with Multiple Versions ..................................................507
16.8.3 rmdel and comb: Removing and Combining Versions ...................509
16.8.4 Using Identification Keywords ....................................................509
16.9 Controlling Access to SCCS ................................................................510
16.9.1 Controlling User Access .............................................................510
16.9.2 Controlling Releases ..................................................................511
16.10 The Revision Control System (RCS) ...................................................511
16.10.1 Creating an RCS File with ci ....................................................511
16.10.2 co and ci: Checking Out and In ..............................................512
16.10.3 Specifying Revisions ................................................................513
16.10.4 rlog and rcs: Other RCS Functions .........................................514
16.11 The Concurrent Version System (CVS) ..............................................517
16.11.1 Setting Up the Repository ........................................................518
16.11.2 Importing Files ........................................................................518
16.11.3 Checking Out .........................................................................519
16.11.4 Editing and Checking In ..........................................................519
16.11.5 Merging and Resolving Conflicts ..............................................520
16.11.6 Setting Tags and Exporting the Repository ................................521
16.11.7 Other Features........................................................................521
16.12 Debugging Programs with gdb .........................................................522
16.12.1 Starting gdb ...........................................................................525
16.12.2 Using a Breakpoint .................................................................526

das76205_fm_i-xI.indd xxiii

12/14/11 4:36 PM

xxiv

Your UNIX/Linux: The Ultimate Guide
16.12.3 Stepping Through the Program ................................................528
16.12.4 Making a Reassignment ..........................................................528
16.12.5 Setting a Watchpoint ..............................................................529
16.12.6 Other Features of gdb ..............................................................531
Summary ....................................................................................................531
Self-Test ......................................................................................................532
Exercises .....................................................................................................533

Chapter 17

Systems Programming I—Files .............................................................535
17.1 System Call Basics ...............................................................................536
17.1.1 Anatomy of a System Call .........................................................536
17.1.2 System Calls vs. Library Functions ..............................................537
17.2 errno and perror: Handling Errors......................................................537
17.3 open: Opening and Creating a File ......................................................539
17.4 close: Closing a File ...........................................................................541
17.5 read: Reading a File ............................................................................542
17.6 write: Writing a File............................................................................543
17.7 ccp.c: Copying a File ..........................................................................543
17.8 lseek: Positioning the Offset Pointer...................................................545
17.8.1 reverse_read.c: Reading a File in Reverse .................................547
17.9 truncate and ftruncate: Truncating a File ..........................................548
17.10 umask: Modifying File Permissions During Creation ...........................549
17.11 Directory Navigation ........................................................................550
17.12 Reading a Directory ..........................................................................552
17.12.1 lls.c: Listing Filenames in a Directory .....................................553
17.13 Modifying Entries in Directory ..........................................................554
17.13.1 mkdir and rmdir: Creating and Removing Directories ...............554
17.13.2 link and symlink: Creating a Hard and Symbolic Link .............554
17.13.3 unlink: Removing a Link .........................................................555
17.13.4 rename: Renaming a File, Directory, or Symbolic Link .................555
17.14 Reading the Inode: struct stat and stat ........................................556
17.14.1 attributes.c: Displaying Some File Attributes .........................557
17.14.2 S_IFMT: Manipulating the st_mode Member .............................558
17.14.3 Using the S_ISxxx Macros to Determine File Type .....................559
17.14.4 Accessing the Permission Bits ...................................................559
17.14.5 lsdir.c: Listing Only Directories .............................................560

das76205_fm_i-xI.indd xxiv

12/14/11 4:36 PM

Contents

xxv
17.15 access: Checking the Real User’s Permissions ....................................560
17.16 Modifying File Attributes...................................................................563
17.16.1 chmod and fchmod: Changing File Permissions ...........................563
17.16.2 chown: Changing Ownership ...................................................564
17.16.3 utime: Changing the Time Stamps...........................................564
17.16.4 atimemtime.c: Creating a File with Identical Time Stamps.........565
Summary ....................................................................................................566
Self-Test ......................................................................................................567
Exercises .....................................................................................................568

Chapter 18

Systems Programming II—Process Control ........................................570
18.1 The Process Revisited ..........................................................................571
18.1.1 The Virtual Address Space .........................................................571
18.1.2 The Process Table .....................................................................573
18.2 The Process Environment ....................................................................573
18.2.1 process.c: Looking Up Some Process Credentials .......................574
18.3 fork: Replicating the Current Process .................................................575
18.3.1 fork.c: A Simple Child Creation Program ...................................575
18.4 exec: The Final Step in Process Creation .............................................577
18.4.1 execl and execv: The Key Members ..........................................578
18.4.2 The Other exec Members...........................................................579
18.5 Gathering the Exit Status ....................................................................580
18.5.1 wait: When the Parent Waits ....................................................580
18.5.2 waitpid: A More Powerful Waiting Mechanism ...........................581
18.6 fork_exec_wait.c: Using All Three Calls .............................................582
18.7 File Sharing.........................................................................................584
18.7.1 The File Descriptor Table............................................................584
18.7.2 The File Table ...........................................................................585
18.7.3 The Vnode Table .......................................................................585
18.7.4 When the Linkage Changes .......................................................585
18.8 File Descriptor Manipulation ...............................................................587
18.8.1 dup and dup2: Duplicating a File Descriptor ................................587
18.8.2 redirection.c: Redirection at Last............................................588
18.8.3 fcntl: Recommended over dup and dup2 ...................................590
18.9 Signal Handling ..................................................................................590
18.9.1 The System Calls.......................................................................592

das76205_fm_i-xI.indd xxv

12/14/11 4:36 PM

xxvi

Your UNIX/Linux: The Ultimate Guide
18.10 sigaction: Installing a Signal Handler ...............................................592
18.10.1 signal.c: Catching the SIGALRM Signal ...................................594
18.10.2 Catching Terminal-Generated Signals.......................................595
18.11 killprocess.c: Using fork-exec-wait and SIGCHLD ............................597
18.12 IPC with Unnamed Pipes ..................................................................599
18.12.1 Using pipe with fork ..............................................................600
18.12.2 pipe2.c: Running UNIX Commands in a Pipe...........................600
Summary ....................................................................................................602
Self-Test ......................................................................................................603
Exercises .....................................................................................................604

PART III

System Administration ..........................................................................607

Chapter 19

System Administration ..........................................................................609
19.1 root: The System Administrator’s Login...............................................610
19.1.1 su: Acquiring Superuser Status...................................................610
19.2 The Administrator’s Privileges .............................................................611
19.3 User Management ..............................................................................612
19.3.1 Understanding /etc/group .......................................................612
19.3.2 Understanding /etc/passwd .....................................................613
19.3.3 Adding and Modifying a User Profile ..........................................614
19.4 Maintaining Security...........................................................................615
19.4.1 Restricted Shell .........................................................................615
19.4.2 Set-User-Id (SUID): Power for a Moment ....................................616
19.4.3 The Sticky Bit ............................................................................617
19.5 Booting and Shutdown.......................................................................618
19.5.1 Booting ....................................................................................618
19.5.2 Shutdown ................................................................................619
19.6 How init Controls the System ...........................................................619
19.6.1 How init Creates the Shell .......................................................620
19.6.2 The rc Scripts ............................................................................621
19.7 Device Files .........................................................................................622
19.7.1 Block and Character Devices ......................................................623
19.7.2 Major and Minor Numbers ........................................................623
19.8 File Systems ........................................................................................624
19.8.1 File System Components ............................................................624
19.8.2 The Standard File Systems and Their Types .................................625

das76205_fm_i-xI.indd xxvi

12/14/11 4:36 PM

xxvii

Contents

19.9 Mounting and Unmounting File Systems ............................................626
19.9.1 mount: Mounting File Systems ....................................................626
19.9.2 umount: Unmounting File Systems ..............................................627
19.10 fsck: File System Checking ...............................................................627
19.11 Managing Disk Space .......................................................................628
19.11.1 df: Reporting Free Space .........................................................629
19.11.2 du: Disk Usage........................................................................629
19.11.3 find Revisited: The Administrator’s Tool .....................................630
19.12 Handling Floppy Diskettes ................................................................630
19.12.1 format and fdformat: Formatting Diskettes .............................631

Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix

das76205_fm_i-xI.indd xxvii

A
B
C
D
E
F
G
H
I

19.12.2 dd: Copying Diskettes ..............................................................631
19.12.3 Handling DOS Diskettes ..........................................................631
19.13 tar: Backing Up Files ........................................................................633
19.13.1 Backing Up Files (-c) ...............................................................633
19.13.2 Restoring Files (-x) ..................................................................634
19.13.3 Displaying the Archive (-t) ......................................................635
19.13.4 Other Options ........................................................................635
19.14 Installing Programs with rpm .............................................................636
19.14.1 Installing and Upgrading Packages ..........................................636
19.14.2 Removing Packages ................................................................637
19.14.3 Querying Packages .................................................................638
Summary ....................................................................................................639
Self-Test ......................................................................................................640
Exercises .....................................................................................................641
The C Shell—Programming Constructs ...............................................643
The Korn and Bash Shells—Exclusive Programming Constructs.....650
The GNU emacs Editor .............................................................................657
vi/vim Command Reference ..................................................................686
The Regular Expression Superset .........................................................693
The HOWTO .............................................................................................696
The ASCII Character Set.........................................................................701
Glossary....................................................................................................705
Solutions to Self-Test Questions...........................................................728
Index.........................................................................................................752

12/14/11 4:36 PM

das76205_fm_i-xI.indd xxviii

12/14/11 4:36 PM

List of Tables

2.1 Organization of the man Documentation ...............................................34
2.2 Escape Sequences Used by echo and printf ...........................................38
2.3 Internal Commands Used by mailx .........................................................43
2.4 stty Settings and Keyboard Commands to Try When Things Go Wrong ......49
3.1 Options to ls .........................................................................................70
3.2 Usage of cp, rm, and mv Commands ........................................................74
3.3 Internal Commands of more and less .....................................................77
4.1 Abbreviations Used by chmod ..................................................................97
4.2 chmod Usage ...........................................................................................99
4.3 Effect of umask Settings on Default Permissions .....................................103
4.4 Major Expressions Used by find (Meaning gets reversed when
– is replaced by +, and vice versa) .........................................................115
5.1 Input Mode Commands .......................................................................124
5.2 Save and Exit Commands of the ex Mode ............................................131
5.3 Navigation and Scrolling Commands....................................................135
5.4 vi Commands Required to Correct Program in Fig. 5.17 ......................141
5.5 A Few Specimen Operator-Command Combinations ............................142
5.6 Search and Replace Commands ............................................................147
5.7 set Options ..........................................................................................150
6.1 The Shell’s Wild Cards and Application .................................................159
7.1 Options to ps .......................................................................................191
7.2 List of Commonly Used Signals .............................................................199
7.3 Job Control Commands ........................................................................203
7.4 Sample crontab Entries (First five fields only) ........................................209

xxix

das76205_fm_i-xI.indd xxix

12/14/11 4:36 PM

xxx

Your UNIX: The Ultimate Guide
8.1 Common Environment Variables ...........................................................217
8.2 Useful Aliases ........................................................................................221
8.3 The History Functions ...........................................................................222
8.4 Shell Settings with set -o .....................................................................228
8.5 Comparative Features of the Shells .......................................................237
9.1 Options to the pr Command ................................................................243
9.2 sort Options ........................................................................................252
10.1 grep Options ......................................................................................268
10.2 The Basic Regular Expression (BRE) Character Set Used by
grep, sed, and awk ..............................................................................272
10.3 The Extended Regular Expression (ERE) Set Used by
grep, egrep and awk....................................................................................276
10.4 Internal Commands Used by sed ........................................................278
10.5 sed Options ........................................................................................280
10.6 Other Basic Regular Expressions (BREs) Used by grep and sed .............287
11.1 The Internet Domains (TLDs and ccTLDs) ...........................................300
11.2 Server and Client Programs ................................................................302
12.1 Arithmetic Operators Used by awk and perl ........................................335
12.2 Assignment Operators (i = 5 initially; result used as
initial value by next line)....................................................................... 336
12.3 The Comparison and Logical Operators ..............................................338
12.4 Matching Regular Expressions.............................................................341
12.5 Built-In Variables .................................................................................346
12.6 Built-in Functions ................................................................................348
13.1 Special Parameters Used by the Shell ..................................................364
13.2 Numerical Comparison Operators Used with test ..............................367
13.3 String Tests with test .........................................................................369
13.4 File Attribute Testing with test ...........................................................370
14.1 Special Escape Sequences Used in Double-Quoted Strings
(See also Table 2.2) .............................................................................407
14.2 The Comparison and Concatenation Operators ..................................408
14.3 Additional Regular Expression Sequences Used by perl ......................413
15.1 Essential Format Specifiers Used by printf and scanf .........................445
15.2 Operator Precedence and Associativity ...............................................461
16.1 SCCS and RCS: Command Usage (File: foo.c) ....................................512
16.2 CVS Keywords (Used with the cvs command) ....................................517
16.3 gdb Commands ..................................................................................523

das76205_fm_i-xI.indd xxx

12/14/11 4:36 PM

List of Tables

xxxi
17.1 Significance of Symbolic Constants Associated with errno ..................539
17.2 The S_ISxxx Macros ............................................................................559
18.1 Signals and Default Disposition ...........................................................591
19.1 The Family of DOS Commands (Linux command name
in parentheses)...................................................................................632
19.2 tar Options ........................................................................................634

das76205_fm_i-xI.indd xxxi

12/14/11 4:36 PM

das76205_fm_i-xI.indd xxxii

12/14/11 4:36 PM

Preface

A language is not worth knowing unless it teaches you to think differently.
—Larry Wall (the creator of Perl) and Randal Schwartz

I

decided to write a book on UNIX without having previously taught the subject. I didn’t like any of the books then available in the market, and there were
hardly any experts whom I could turn to for help. Having learned the subject the hard
way, I decided to present my experience in the form of a book, but without knowing
its intended audience. When I taught the subject many years later, I found the answers
to the questions posed by students to be already in the book. I discovered I was on the
right track and that I had actually written a textbook.
UNIX has survived the test of time and is likely to do so for some time to come. Yet
UNIX is still described by many as “unfriendly” and “unforgiving.” Beginners continue to
feel overwhelmed by its apparent complexity. Even experienced computer professionals
have been unable to embrace the paradigm of doing work using the command line and
its myriad options and complex syntaxes. All UNIX and Linux systems today offer the
graphical user interface (GUI), but the command line remains the backbone of the system.
The excitement that UNIX generates lies in the fact that many of its powers are
hidden. UNIX doesn’t offer everything on a platter; it encourages you to create and innovate. Figuring out a command combination or designing a script that does a complex
job is a real challenge to the UNIX enthusiast. This is what UNIX is, and it had better
remain that way. If you appreciate that there is a method to this madness, then you are
on the right track and this book is meant for you.

How This Book Is Different
Facing a UNIX box was my first encounter with computers, and prolonged periods of
struggle with the system have led me to believe that the stumbling blocks to understanding UNIX are often different from what they are perceived to be. It’s not that UNIX is
difficult, but that its authors have not been imaginative enough to make it appear otherwise. Books often present, say, 20 options to a command even though only a couple of
them could actually be useful. It’s frustration that led me to develop my version of the
“true” UNIX book—a book that knows what users actually need and one they would
like to have with them all the time.
xxxiii

das76205_fm_i-xI.indd xxxiii

12/14/11 4:36 PM

Your UNIX/Linux: The Ultimate Guide

xxxiv

Real-Life Examples UNIX concepts are simple, but they are also abstract, and it’s
often not obvious why a certain feature is handled in a particular way. The mastery of
this operating system requires a clear understanding of these concepts. I have made sure
that the key features are explained clearly to reveal both their design considerations and
their relevance in the real world. You’ll find that many examples of this text refer to
real-life situations.
Both a User’s and Programmer’s Guide There are mainly two categories of UNIX
users: those who use its native tools, and others who develop tools using shell scripts and
the UNIX system call library. This book—probably the only one of its kind—adequately
addresses both of these segments but has a size that is convenient to read and carry.
The “user” category is served by the first 11 chapters, which is adequate for an
introductory UNIX course. The “developer” is a shell or systems programmer who also
needs to know how things work, say, how a directory is affected when a file is created
or linked. For their benefit, the initial chapters contain special boxes that probe key
concepts. This arrangement shouldn’t affect the beginner, who may quietly ignore these
portions. UNIX shines through Chapters 16, 17, and 18, so these chapters are compulsory
reading for systems programmers.
Strong Learning Aids The pedagogical aids are a strong feature of this book. They
have various names, for example, Note, Caution, and Tip. I consider Linux to be an
important member of the UNIX family, so I have separately highlighted Linux features
using the penguin as identifier.
I don’t agree with the approach adopted by many authors of treating each shell in
a separate chapter. Instead, I have discussed key concepts using mainly the Bash shell.
Deviations are addressed by separate asides for the C shell, Korn, and Bourne shells.
Numerous Questions and Exercises This book features an enormous number of
questions that test the reader’s knowledge. More than a third of them are Self-Test questions, and their answers are provided in Appendix I. These questions are all targeted
toward beginners who will do well to answer them before moving on to the next chapter.
More rigorous and extensive questioning is reserved for the Exercises sections.
Some of them pose real challenges, and it may take you some time to solve them. These
exercises reinforce and often add to your knowledge of UNIX, so don’t ignore them.
The answers to these questions are available to adopters of the book at the book’s Web
site, http://www.mhhe.com/das.

What’s New In This Edition
Some of the reviewers seemed to be quite comfortable with the second edition as it is,
and had warned that the improvement may not be “very productive.” Keeping this view
in mind, I have made the following changes in this edition:
• While the chapter on vi/vim has been retained, the one on emacs has been
condensed and relegated to an appendix. To make the transition to vi easier for
beginners, Chapter 3 features the pico editor.

das76205_fm_i-xI.indd xxxiv

12/14/11 4:36 PM

xxxv

Preface

• The bc calculator utility makes a return in this edition, but only as an efficient
computing tool for shell programmers.
• A separate chapter on the essentials of C programming has been added. The treatment, though brief, is just adequate to understand the two chapters on systems
programming that follow.
• Chapter 15 now includes the popular Concurrent Version System (CVS), which is
found on all Linux systems. SCCS and RCS continue to find place in this edition,
but at least one of them might be removed in the next edition.
• The GNU debugger (gdb) has been included in this edition since it is superior to
dbx, even though the latter has been retained.
These changes have resulted in a nominal increase in the size of the book. In spite of
increased coverage on Linux and GNU tools, the generic character of the book has
been retained; it doesn’t focus on any particular flavor of UNIX, but variations found
in Solaris and Linux have been highlighted.

Understanding the Organization
This edition is logically divided into user and programming sections. Essential UNIX is
confined to the first 11 chapters, which culminate with a discussion on networking tools.
Programming material comprising awk, shell programming, systems programming, and
perl are covered in the the next seven chapters. The final chapter presents the essentials
of system administration.
Introducing UNIX Chapter 1 reveals the key UNIX concepts through a simple hands-on
session. This is followed by a brief history and presentation of the features of UNIX. You
are introduced to the kernel and shell, who between them, handle the system’s workload.
You will also understand the role played by standards bodies like POSIX and The Open
Group in building the framework for developing portable applications.
Chapter 2 presents the structure of the UNIX command line. It also discusses the
techniques of using the man command to look up the online documentation. You learn
to use an email program, change your password and see what’s going on in the system.
Things can and will go wrong, so you also need to know how to use the keyboard for
corrective action.
Files The file is one of the two pillars that support UNIX, and the next three chapters
discuss files. Chapter 3 discusses the various types of files you’ll find on your system
and the commands that handle them. You’ll learn to create directories, navigate a directory structure, and copy and delete files in a directory. You’ll also learn to edit a file
with the pico editor. UNIX also offers a host of compression utilities that you need to
use to conserve disk space.
Files have attributes (properties), and Chapter 4 presents the major attributes, especially the ones displayed by the ls -l command. Be aware that your files and directories
are open to attack, so learn to protect them by manipulating their permissions. Use links
to access a file by multiple names. You’ll also forget where you have kept your files, so
you need to be familiar with the find command.

das76205_fm_i-xI.indd xxxv

12/14/11 4:36 PM

xxxvi

Your UNIX/Linux: The Ultimate Guide
How productive you eventually are also depends on how well you exploit the
features of your editor. Chapter 5 presents vi, one of the most powerful text editors
found in any operating environment. A programmer probably uses the editor more
than anyone else, so most examples in this chapter use snippets of program code.
Appendix D presents a summary of the features of vi.
The Shell and Process You now need to understand a very important program that is
constantly interacting with you—the shell. Chapter 6 presents the interpretive features
of the shell, including many of its metacharacters. Learn to use wild cards to match
a group of similar filenames with a single pattern. Manipulate the input and output of
commands using redirection and pipes. The shell is also a programming language, so
you have to wait until Chapter 13 to understand it completely.
Chapter 7 introduces the process as the other pillar of the UNIX system. Processes
are similar to files, and processes also have attributes. Understand how the fork-exec
mechanism is used to create a process. Learn to control processes, move them between
foreground and background, and also kill them by sending signals.
The UNIX shell provides excellent opportunities to customize your environment
(Chapter 8). Understand and manipulate shell variables, create command aliases and
use the history mechanism to recall, edit and re-execute previous commands. Choose a
suitable shell that offers all of these features and learn to use the initialization scripts to
save the changes you’ve made to the environment.
Filters The next two chapters deal with filters—those special commands in the UNIX
tool kit that handle all text manipulation tasks. Chapter 9 presents the simple ones and
shows how they are most effective when they are connected to one another. A special
examples section features three real-life applications that are handled by these filters
working in pipelines.
Chapter 10 discusses two powerful filters—grep and sed—that, between them,
handle all pattern search, edit, and replace operations. At this stage, you’ll be introduced
to regular expressions, an elaborate pattern-matching mechanism that often makes
searching and replacement a lot easier. Filters are followed by a chapter on networking
tools (Chapter 11), which concludes the first part of this book.
Programming The next seven chapters handle most of the programming features of
UNIX. The awk command makes its appearance as a filter and a programming language
in Chapter 12. Knowing awk and its standard programming constructs (like the if,
for, and while constructs) should prepare you well for shell programming, perl, and
C programming.
Eventually, you’ll place all of your commands and pipelines in shell scripts. Use
the programming features of the shell discussed in Chapter 13 to develop both interactive
and noninteractive scripts. Learn to design a script whose behavior depends on the name
by which it is invoked. The three sample scripts featured in the chapter are compulsory
reading for a shell programmer.
The next four chapters are directly or indirectly related to C programming.
Chapter 14 presents a primer on C programming, the only new chapter in this edition.

das76205_fm_i-xI.indd xxxvi

12/14/11 4:36 PM

xxxvii

Preface

This is followed by the program development tools (Chapter 15). Use the make utility
and a powerful debugger (gdb) for managing and debugging programs. Also, learn to
maintain multiple versions of a program using SCCS, RCS, and CVS.
Chapter 16 is the first of two chapters that feature the use of system calls in the C
programming environment. This chapter discusses the system calls related to files and
I/O. Write programs that perform directory-oriented functions like listing files. Also
learn to fetch and manipulate file attributes stored in the inode.
Chapter 17 discusses the system calls related to processes. Learn to create processes
using the fork and exec family of system calls. Once you’ve understood how the kernel
maintains the metadata of an open file in memory, you’ll be able to implement both
redirection and pipelines and to handle signals in your programs.
We encounter perl in Chapter 18 as the most powerful filter and scripting language in the UNIX world. Most UNIX concepts are embedded in the design of perl,
the reason why many UNIX users can’t do without it. Even though we can’t do justice
to perl in a single chapter, Chapter 18 represents a useful beginning.
Finally, every user must know the routine tasks related to system administration, and
Chapter 19 addresses the basic issues in this domain. Understand the important security
features provided by the system. Be familiar with the activities associated with system
startup and shutdown, and how file systems are mounted and checked for consistency.
Also learn to do some elementary backups.

Acknowledgments
Every edition of this book has provided me with an opportunity to work with a new team.
The prime mover for this project is Raghu Srinivasan, the global publisher, and it was a
delight to work with him and Melissa Leick, the project manager. I am impressed with
the way they have guided the team at McGraw-Hill. I must also thank Brenda Rolwes,
the design coordinator, and Curt Reynolds for the marketing arrangements that he was
responsible for. Special mention must be made of Melinda Bilecki, the developmental
editor, who remains as agile as ever. She is the only person associated with the book
since its conception, and this continuity has benefitted me immensely.
Credit must also go to the reviewers who have played a significant role in shaping
the form and content of this edition:
Ivan Bajic, Sas Diego State University
Bruce Elenbogen, University of Michigan–Dearborn
Ziad Kobti, University of Windsor
Galina Piatnitskaia, University of Missouri–St. Louis
Paul Piwowarski, University of Kentucky
Darren Provine, Rowan University
Quazi M. Rahman, The University of Western Ontario
I am thankful to Deepti Narwat, the project manager at Cenveo Publisher Services,
who handled the entire production process with confidence and the utmost sincerity.
There have been many others who couldn’t be mentioned by name, but have contributed
just the same.

das76205_fm_i-xI.indd xxxvii

12/14/11 4:36 PM

Your UNIX/Linux: The Ultimate Guide

xxxviii

Final Words of “Wisdom”
Most of the examples have been tested on Solaris and Linux, but I can’t guarantee that
they will run error-free on every system. UNIX fragmentation makes sweeping generalizations virtually impossible. If some commands don’t work in the way specified in
this text, don’t conclude that the system has bugs. Nevertheless, bugs in these examples
are still possible, and I welcome ones that you may hit upon.
Before I take leave, a note of caution would be in order. Many people missed the
UNIX bus through confused and misguided thinking and are now regretting it. Let this
not happen to you. Once you have decided to exploit UNIX, you’ll learn to build on
what’s already provided without reinventing the wheel. Sooner rather than later, you’ll
find a world of opportunity and excitement opening up. Approach the subject with zeal
and confidence; I am with you.
Sumitabha Das

das76205_fm_i-xI.indd xxxviii

12/14/11 4:36 PM

Conventions Used in This Book
The key terms used in the book (like regular expression) are shown in a bold font. Apart
from this, the following conventions have been used in this book:
• Commands, internal commands and user input in examples are shown in bold
constant width font:
Many commands in more including f and b use a repeat factor.
The shell features three types of loops—while, until, and for.
Enter your name: henry
• Apart from command output, filenames, strings, symbols, expressions, options,
and keywords are shown in constant width font. For example:
Most commands are located in /bin and /usr/bin.
Try doing that with the name gordon lightfoot.
Use the expression wilco[cx]k*s* with the -l option.
The shell looks for the characters >, < and << in the command line.
The -mtime keyword looks for the modification time of a file.
• Machine and domain names, email addresses, newsgroups, and URLs are displayed
in italics:
When henry logs on to the machine uranus ......
User henry on this host can be addressed as henry@calcs.planets.com.
The newsgroup comp.lang.perl discusses problems related to perl.
Executables for all UNIX flavors are available at http://www.perl.com.
• Place-holders for filenames, terms, and explanatory comments within examples
are displayed in italics:
Use the -f filename option if this doesn’t work.
This process has a controlling terminal.
$ cd ../..

Moves two levels up

The following abbreviations, shortcuts and symbols have been used:
•
•
•
•
•
•
•
•
•
•

das76205_fm_i-xI.indd xxxix

SVR4—System V Release 4
sh—Bourne shell
csh—C shell
ksh—Korn shell
$HOME/flname—The file flname in the home directory
~/flname—The file flname in the home directory
foo, bar, and foobar—Generic file and directory names as used on Usenet
.... for lines that are not shown
This box indicates the space character.
This pair of arrows
indicates the tab character.

12/14/11 4:36 PM

das76205_fm_i-xI.indd xl

12/14/11 4:36 PM

PA R T

I

UNIX for the User

das76205_Ch01_001-024.indd 1

12/13/11 10:19 AM

das76205_Ch01_001-024.indd 2

12/13/11 10:19 AM

CHAPTER

1

Introducing UNIX

I

n this opening chapter, we commence our journey into the world of UNIX. We’ll
discover why a computer needs an operating system and how UNIX more than
fulfills that requirement. Through a hands-on session, we’ll learn to play with the UNIX
system. We’ll use the tools UNIX provides to perform the basic file and directory handling
operations. We’ll also have a glimpse of the process that makes a program run on UNIX.
As we absorb this knowledge, we’ll place it against the rather turbulent background that UNIX had to grow through. We’ll learn how contributions from different
sources led to both the enrichment and fragmentation of UNIX. Knowledge of the design
considerations will also help us understand why UNIX sometimes behaves in a seemingly awkward manner. We’ll examine the UNIX architecture and understand how two
agencies (the kernel and shell) between themselves handle all the work of the system.

Objectives
•
•
•
•
•
•
•
•
•
•
•
•

1.1

Learn what an operating system is and how UNIX is different from other systems.
Understand the role of the system administrator.
Log in and out of a UNIX system.
Run a few commands that report on the system.
Use more commands to view processes and handle files and directories.
Find out how UNIX got steadily fragmented by the emergence of other flavors.
Understand the merging of POSIX and the Single UNIX Specification into a single
UNIX standard.
Learn about the emergence of Linux as a strong, viable and free alternative.
Discover the UNIX architecture that includes the kernel and shell.
Discover the two key features—the file and process—that UNIX rests on.
Know the role of system calls in making programs work.
Learn the “do-one-thing-well” philosophy that UNIX uses to solve complex problems.

The Operating System
Computers are designed to run programs. But a program can run only if the computer
it is running on has some basic intelligence to begin with. This intelligence allocates
memory for the program, runs each program instruction on the CPU, and accesses the

3

das76205_Ch01_001-024.indd 3

12/13/11 10:19 AM

4

Your UNIX/Linux: The Ultimate Guide
hardware on behalf of the program. A special piece of preinstalled software performs
this job. This software is known as the computer’s operating system.
An operating system is the software that manages the computer’s hardware and
provides a convenient and safe environment for running programs. It acts as an interface
between programs and the hardware resources that these programs access (like memory,
hard disk and printer). It is loaded into memory when a computer is booted and remains
active as long as the machine is up.
To grasp the key features of an operating system, let’s consider the management
tasks it has to perform when we run a program. These operations also depend on the
operating system we are using, but the following actions are common to most systems:
• The operating system allocates memory for the program and loads the program to
the allocated memory.
• It also loads the CPU registers with control information related to the program. The
registers maintain the memory locations where each segment of a program is stored.
• The instructions provided in the program are executed by the CPU. The operating
system keeps track of the instruction that was last executed. This enables it to resume a program if it had to be taken out of the CPU before it completed execution.
• If the program needs to access the hardware, it makes a call to the operating system
rather than attempting to do the job itself. For instance, if the program needs to
read a file on disk, the operating system directs the disk controller to open the file
and make the data available to the program.
• After the program has completed execution, the operating system cleans up the
memory and registers and makes them available for the next program.
Modern operating systems are multiprogramming, i.e., they allow multiple programs to reside in memory. However, on computers with a single CPU, only one program
can run at one time. Rather than allow a single program to run to completion without
interruption, an operating system generally allows a program to run for an instant, saves
its current state and then loads the next program in the queue. The operating system
creates a process for each program and then controls the switching of these processes.
Most programs often access the disk or the terminal to read or write data. These
I/O operations keep the CPU idle, so the operating system takes the program out of the
CPU while the I/O operation is in progress. It then schedules another program to run.
The previous program can resume execution only after the I/O operation completes. This
ensures maximum utilization of the CPU.
In addition to these basic services, operating systems provide a wide range of
services—from creating files and directories to copying files across a network and performing backups. These tools are often standalone programs that don’t form the core
of the operating system, but they may be considered as additional services that benefit
both users, programmers, and system administrators.
Knowing the functions performed by an operating system and the way they are
implemented on your computer often helps you write better programs. True, a lot can
be done without knowing the operating system, but a UNIX professional needs to look
beyond the big picture to discover how things actually work.

das76205_Ch01_001-024.indd 4

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

5

In a multiprogramming environment, the operating system has to ensure that a process performing an I/O operation doesn’t hold up the CPU. It must schedule another process while
the I/O operation is in progress. The previous process is said to block, i.e., wait for the event to
complete. We’ll often use this term in this text to refer to this state of a process.

Note

1.2

The UNIX Operating System
There have been many operating systems in the past, one at least from each hardware
vendor. They were written in a near-machine language known as assembler. The operating
systems were proprietary because assembler code developed on one machine wouldn’t
run on another. Vendors required consumers to purchase expensive proprietary hardware
and software if two dissimilar machines needed to talk to each other. Ken Thompson
and Dennis Ritchie changed all that forever. They created UNIX.
The UNIX operating system marks a strong departure from tradition. It has practically everything an operating system should have, but it also introduces a number of
concepts previously unknown to the computing community. Beginners with some experience in Windows think of UNIX in terms of Windows, quite oblivious to the fact that
the similarities are only superficial. UNIX is way ahead of other systems in sheer power.
UNIX is not written in assembler but in C. C is a high-level language that was
designed with portability considerations in mind, which explains why UNIX systems are
available on practically every hardware platform. Apart from handling the basic operating
system functions, UNIX offers a host of applications that benefit users, programmers,
and system administrators. It encourages users to combine multiple programs to solve
a complex problem. For programmers, UNIX offers a rich set of programming tools
that aid in developing, debugging, and maintaining programs. UNIX is also more easily
maintained than most systems.
One of these programs is the system’s command interpreter, called the shell. You
interact with a UNIX system through the shell. Key in a word, and the shell interprets
it as a command to be executed. A command may already exist on the system as one of
several hundred native tools, or it could be one written by you. However, as mentioned,
the power of UNIX lies in combining these commands in the same way the English
language lets you combine words to generate a meaningful idea. As you walk through
the chapters of the text, you’ll soon discover that this is a major strength of the system.
UNIX was written by programmers for their own use, so things that appear obvious to them don’t always appear obvious to us. However, that doesn’t imply that UNIX
is unconquerable; in fact, it’s great fun. If you are willing to put in some guided effort,
you’ll gradually see the UNIX story unfold with clarity and simplicity. Focus your
attention on the essentials, and try to understand the designers’ minds and objectives.
Even though UNIX sometimes appears unfriendly, it in fact challenges you to unravel
its mysteries. In this book, we take up the challenge.

Note

das76205_Ch01_001-024.indd 5

The UNIX system doesn’t offer a fixed set of services. In fact, you have to use your imagination
in devising improvised tools from the existing ones. This is what makes UNIX so challenging
and exciting.

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

6

1.3

Knowing Your Machine
Unlike Windows, UNIX can be used by several users concurrently. In other words, a
single copy of the operating system software installed on just one machine can serve the
needs of hundreds of users. These users could use dumb terminals or their own desktop
PCs to access a central UNIX computer. This computer will probably be located in a
separate room with restricted access. If you are using a PC, then it must be configured
properly before it can be used to connect to a UNIX system.
Things are quite different, however, when you are using a workstation. This is a
computer capable of producing high-quality graphics but meant to be used by a single
user. Unlike the dumb terminal, a workstation has its own CPU, memory (the RAM—
random access memory), hard disk, DVD-ROM, mouse as a pointing device, and printer.
Since it has all the things that UNIX needs, a workstation can run its own UNIX. Desktop
PCs are also often referred to as workstations because there are versions of UNIX (like
Linux) that can run on them.
Even though workstations and PCs run UNIX and can be used in standalone mode,
they are often connected to a larger, more powerful computer in the same way terminals
are. There are a number of reasons you might want such an arrangement:
• The central computer is administered properly, and you might want to keep all of
your valuable files there so they are backed up regularly.
• You might want to use a powerful program that your workstation doesn’t have but
the central computer does.
• All of your incoming and outgoing mail is handled by the central machine, which
may be your only link with the outside world, i.e., the Internet.
Every workstation and PC provides a terminal emulation facility that makes it
abandon its normal mode and behave like a simple dumb terminal instead. The workstation then doesn’t use its own hard disk, CPU, or memory for doing any work except
providing the minimal resources required by the terminal emulation software. The terminal emulation facility enables you to run a program on a remote machine using the
remote machine’s memory and CPU, and not your own. Create a file in this mode, and
the file is saved on the remote machine’s hard disk.
When you press a key, the computer generates a character which represents the
smallest piece of information that you can deal with. It could be a letter, number, symbol,
or control sequence (like [Ctrl-f]). The string 10:20 pm contains eight characters (one for
the space). Every character is associated with a unique ASCII value (ASCII—American
Standard Code for Information Interchange). The letter A has the ASCII value of 65; the
bang (!) has the value of 33. Both take up one byte (eight bits) each on your computer.
Many UNIX programs make use of these ASCII values.
Even though you may be completely familiar with the keyboard of your Windows
PC, note that the functions of many of the keys change when the same PC doubles as a
UNIX or Linux box. The [F1] key doesn’t invoke help, and the [Delete] key may not
always delete characters. Moreover, every key has some use in UNIX. When you see
a symbol like ` (backquote) used in this book, you must be able to locate it easily on
your keyboard (on the top-left), and not confuse it with the ‘ (single quote), because
they have totally different functions.

das76205_Ch01_001-024.indd 6

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

7

Using a combination of [Alt] and a function key, you can have multiple virtual console
or terminal sessions on a single PC. You can log in several times to the same computer,
with a separate “terminal” for each session. A single screen is shared by all sessions,
which are accessed by using [Alt][F1], [Alt][F2], and so on.

Linux

1.4

The System Administrator
On a large system serving hundreds of users, someone has to be in charge of administration of the system. This person is known as the system administrator. The administrator is responsible for the management of the entire setup. She allocates user accounts,
maintains file systems, takes backups, manages disk space, and performs several other
important functions. She is the person to be contacted in case of a genuine problem.
If you own a workstation or PC that runs some flavor of UNIX, then you are
probably its administrator. You are then directly responsible for its startup, shutdown,
and maintenance. If you lose a file, it’s your job to get it from a backup. If things don’t
work properly, you have to try all possible means to set them right before you decide
to call a maintenance person.
If you are not the administrator, you can use a UNIX machine only after she
has opened an account with a user-id and password for your use. These authentication
parameters are maintained in two separate files on your system. You can’t simply sit
down at any terminal and start banging away unless you first log on to the system using
a valid user-id–password combination.
The administrator uses a special user-id to log on to the system: it is called root.
The root user has near-absolute powers. Some programs can only be run from this
account—for instance, the program that creates the user account itself.

1.5

Logging In and Out
Let’s get down to business and see for ourselves what a UNIX session is really like. A
personal interaction with the system often drives home a point better than the preaching
of textbooks (including this one). In this section, we’ll quickly walk through the procedure of logging in and out of a UNIX box, but first let’s consider the possible options
we have today to connect to a UNIX machine.
The good old dumb terminal connected to the computer’s serial port was once
the only means of connecting to a UNIX system. Later, the TELNET program became
popular for connecting in a network. For security reasons (explained in Chapter 11), the
TELNET facility could be disabled on your system, and the secure shell (SSH) could
be the only means of connecting to a remote UNIX box. In that case, UNIX and Linux
users can use the ssh command, if available. Windows users may use Putty or any of
the free SSH programs available on the Net.

1.5.1

das76205_Ch01_001-024.indd 7

Logging In
We’ll ignore the password-free access that is permitted by the secure shell programs and
consider the situation where the system authenticates you by your response to the login

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

8

and password prompts. Each response should be followed by the [Enter] key. This is
how user romeo gains access to the system:
SunOS 5.8
login: romeo[Enter]
Password: *********[Enter]

A Sun machine running Solaris 8

Password not shown

Note that the password is not shown on the screen for security reasons. The appearance of the login prompt signifies that the system is available for someone to log
in and the previous user has logged out (i.e., finished her work and disconnected). The
prompt here is preceded by SunOS 5.8, the version of the operating system in Solaris
8, the flavor of UNIX offered by Sun. Your system could show a different string here
(if at all). If you make a mistake, this is what you could be seeing:
Login incorrect
login:

This simply tells us that either the user-id or password (or both) is incorrect. When
you get both of these parameters correct, the system lets you in:
Last login: Thu May 8 06:48:39 from saturn.heavens.com
$ _
The cursor shown by the _ character

The system here displays a message showing the last time you logged in. This is
followed by the prompt string, which here is a $. Your prompt string could be a %, which
is quite popular in the academic world. The system administrator, who uses the root
account to log in, uses # as the prompt. Prompt strings can also be customized. Here’s
one that shows the current directory:
[/home/romeo]

Before we move on, be aware that a program, known as the shell, is now running
at this terminal, waiting to take your input. Whatever we key in now goes as input to
this program. UNIX offers a variety of such programs for you to choose from. The shell
is the command interpreter that interacts both with the user and the operating system.
When the administrator opens a user account, she also sets a specific shell for the user.
Linux systems come preconfigured with informative prompts like this one, which shows
the machine (saturn), username (romeo) and the current directory (/home/romeo):
Linux

romeo@saturn:/home/romeo >

We’ll learn to customize our prompt string in Chapter 8.
1.5.2

Logging Out
Before you try out some of the programs available on the system, you must first know
how to log out. That also depends on the shell you use, so first try
[Ctrl-d]

das76205_Ch01_001-024.indd 8

Keep [Ctrl] pressed and then press d

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

9

If this key sequence doesn’t work but instead shows you the message
Use “logout” to log out, then do as directed:
logout

The shell that we would be mostly using in this book is Bash, and this shell uses
both of the preceding techniques. But there’s a C shell also, which accepts only logout,
and yet another shell named Bourne which works only with [Ctrl-d]. In case neither
technique works, use exit, which works in most cases:
$ exit
login:

System now available for next user

The appearance of the login prompt makes the system available for the next user.
Now log in again so you can try out all the commands featured in the hands-on session
that follows.
Henceforth, we’ll use the terms privileged user, superuser and system administrator to refer to the
root user account that is used by the administrator for logging in, and we’ll use nonprivileged
user and ordinary user to mean all other users. It’s often important to make this distinction
because the root user enjoys certain privileges that are denied others.

Note

1.6

A Hands-On Session
After you have successfully made your entry by providing the user-id and password at
the two prompts, you have free access to the UNIX command set reserved for general
use. When you key in a word, the system interprets it as a command to do something.
In most cases, the command signifies the execution of a program on disk that has the
name you keyed in.
Unlike in DOS, where a command can be run in both lower- and uppercase,
UNIX commands are generally in lowercase. Try using the date command, but use
DATE instead of date:
$ DATE
ksh: DATE: not found

In DOS, both date and DATE would display the same output. In UNIX, however,
there’s no command named DATE (no file named DATE), but there’s one named date. In
UNIX, lowercase is typically used more than uppercase.
We’ll now acquaint ourselves with a few of these commands, which we categorize
into three groups—system information, processes, and files. In this session, we’ll use
them only to get a feel for the system. The commands will be examined in more detail
later in separate chapters.

Caution

das76205_Ch01_001-024.indd 9

Make sure the [CapsLock] key is not permanently set on your machine. When inserting a block
of text in uppercase in editors like pico (3.17) and vi (Chapter 5), we often set this key. When
working at the prompt, however, nothing will work if the terminal is not set to lowercase.

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

10
1.6.1

System Information with date and who
Using date Every UNIX system maintains an internal clock that you can access to print
the current system date and time. UNIX does it with a single command named date:
$ date
Fri Nov 13 09:32:16 GMT 2009
$ _

Prompt returns; you can now enter next command

DATE didn’t work, but date did and returned the prompt. This format is seen in email
messages except that the time zone (here, GMT) could differ on your system. It’s worth
noting that a nonprivileged user runs date with a limited scope; she can’t change the
system date, while the privileged user (i.e. root) can.
Using who to View the List of Current Users UNIX can be used concurrently by
multiple users, and you might be interested in knowing who is using the system when
you are. Use the who command:
$ who
romeo
henry
steve
$ _

console
pts/4
pts/5

Nov 13 9 09:31 (:0)
Nov 13 09:31
(:0.0)
Nov 13 09:32
(saturn.heavens.com)

There are currently three users—romeo, henry, and steve—sharing the CPU. These
names are actually the user-ids they used to log in. The output also includes your own
user-id, romeo, which you entered at the login prompt to gain entry to the system. The
date and time of login are shown in three columns.
Observe that when a command has completed its run, the prompt is returned. The
return of the prompt indicates that all work relating to the previous command has been
completed, and the system is ready to accept the next command. Henceforth, we’ll not
indicate this return except in those situations where the return is significant.
You logged in with the name romeo, so the system addresses you by this name
and associates romeo with whatever work you do. Create a file, and the system will
make romeo the owner of the file. Execute a program, and romeo will be owner of the
process associated with your program. Send mail to another user, and the system will
inform the recipient that mail has arrived from romeo.

Note

1.6.2

das76205_Ch01_001-024.indd 10

UNIX isn’t just a repository of commands producing informative output. You can extract useful information from command output to use with other commands. For instance, you can
extract the day of the week (here, Fri.) from the date output and then devise a program that
does different things depending on the day the program is invoked. You can also “cut” the
user-ids from the who output and use the list with the mailx command to send mail to all users currently logged in. The facility to perform these useful tasks with one or two lines of code
makes UNIX truly different from other operating systems.

Viewing Processes with ps
The process is a key component of any operating system, so let’s run a command that
displays the processes running at our terminal. The ps command does this job, and the
following command shows that currently only one is running:

12/13/11 10:19 AM

11

Chapter 1: Introducing UNIX
$ ps
PID TTY
4249 pts/3
4281 pts/3

TIME CMD
00:00:00 bash
00:00:00 ps

Shell running all the time!

We observed that the shell program is always running at your terminal, and the
ps output bears testimony to the fact. When you run several programs, there will be
multiple lines in the ps output. The last column shows a process named bash, which
represents the Bash shell (an advanced shell that is standard on Linux). This process has
a unique number (4249, called the PID, the process-id) and is killed when you log out.
In fact, the three commands and key sequences recommended for use in logging out in
Section 1.5.2 kill this process.
Even though we are using the Bash shell here, you could be using another shell.
Instead of bash, you could see sh (the primitive Bourne shell), csh (C shell—still
popular today) or ksh (Korn shell). To know the one that is running for you, use the
echo command like this:
$ echo $SHELL
/bin/bash

The Bash shell

$SHELL is one of the several shell variables available on your system. Throughout this
book, we’ll compare the features of these shells and discover features available in one
shell but not in another. If a command doesn’t produce output as explained in this text,
the problem can often be attributed to the shell.
1.6.3

Handling Files
UNIX maintains all data in containers called files. These files are assigned names, and
a group of filenames are held together in another separate file known as a directory. In
this section and in Section 1.6.4, we take a look at some of the basic commands offered
by UNIX to handle files and directories.
Creating a File with echo There are several ways to create a file. Here we use the
echo command with a special symbol (the >):
$ echo date > foo
$ _

No display; prompt returns

The echo command is meant to display a message on the terminal, but here the message
(date) goes to the file foo instead. We’ll not concern ourselves with the role played
by the shell here but simply note that the > is a convenient mechanism of redirecting
command output.
Displaying a File with cat The cat command displays the contents of files, so let’s
use it to view the file that we just created:
$ cat foo
date

Observe that we used both the echo and cat commands with an additional word
(date and foo). They are known as arguments. UNIX commands are often used

das76205_Ch01_001-024.indd 11

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

12

with arguments, and the variety of these arguments make these commands behave in
numerous ways.
Copying a File with cp We now use the cp command to copy the file foo that we
just created:
$ cp foo foo.sh
$ _

No message; prompt returns

Note that cp needs two arguments and operates silently. If you run cat foo.sh now,
cat will also display the string date.
Displaying List of Filenames with ls Now that we have two identical files, we can
produce a list of their names with the ls command:
$ ls
foo

foo.sh

In Chapters 3 and 4, we’ll use the ls command to display the attributes of files and
directories.
Renaming a File with mv The mv command renames a file, and the following sequence
renames foo.sh to foo.shell. We also confirm the action by running ls:
$ mv foo.sh foo.shell
$ ls
foo
foo.shell

Removing a File with rm The rm command deletes files, and this one removes the
file foo.shell:
$ rm foo.shell
$ ls
foo

Only foo is left

ls confirms our action yet again. Observe that cp, rm, and mv behave silently and return
the prompt; they don’t clutter the screen with verbose output. Silence here implies success; the commands worked in the way they were expected to.
1.6.4

das76205_Ch01_001-024.indd 12

Handling Directories
The files foo and foo.sh are ordinary files. Every file has an association with a directory,
and we often describe this association (somewhat loosely) by saying that a file resides in
a directory. A user too is associated with a directory, and this is conveniently expressed
by saying that a user is placed in a directory called the current directory. UNIX considers
a directory as a file, and some commands work with both ordinary files and directories.
The file foo can be considered to reside in the current directory. This directory
was not created by us but by the system administrator when opening an account for

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

13

user romeo. But we can also create a directory, copy a file to it, navigate to that directory, and remove it.
Creating a Directory with mkdir The mkdir command creates a directory. Here the
command creates one named scripts:
$ mkdir scripts
$ _

We now have one file and one directory, both in the current directory. ls will now display
both filenames, but if you follow it with -F (with a space between ls and -F), then you
can identify a directory easily:
$ ls -F
foo

scripts/

-F marks directory with a /

ls here uses an argument that begins with a hyphen, but this argument is appropriately
called an option. Options change the default behavior of a command, and the -F option
modifies the ls output by using the / to mark a directory name.
Copying a File to a Directory The same command in UNIX often works with both
a file and a directory. For instance, the cp command can be used to copy a file to a
directory. Here, cp copies foo to the scripts directory:
$ cp foo scripts
$ _

scripts here is a directory

Directory Navigation with pwd and cd We can know what the current directory is
by using the pwd command and change it by using cd. Before we change our location,
however, let’s use pwd to find out how the command describes the current directory:
$ pwd
/home/romeo

The output shows a pathname, which represents a hierarchy of three directory names.
We are currently stationed in the directory romeo which in turn is below home. The first
/ indicates the top-most directory called root, so home is below the root directory. Don’t
confuse this directory with the root user account.
Since our current directory contains a directory named scripts, we can now use
the cd command to change our location to that directory:
$ cd scripts
$ pwd
/home/romeo/scripts

We have descended one level in this file system hierarchy. This directory should now
contain the file foo that we copied earlier. List this directory with ls, and then remove
foo with rm:

das76205_Ch01_001-024.indd 13

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

14
$ ls
foo
$ rm foo
$ _

The file is gone and the directory is now empty. The directory can now be removed
with rmdir but only after we have moved away from this directory. Let’s return to the
directory we came from before we use rmdir:
$ cd /home/romeo
$ pwd
/home/romeo
$ rmdir scripts
$ ls
foo

We now have a single file left in the current directory. You’ll recall that we had used the
echo command to create this file containing the string date:
$ cat foo
date

We can remove foo with rm, but why not run it like this?
$ sh foo
Fri Nov 13 19:24:26 GMT 2009

This runs the sh command and instructs it to run with foo. Since sh represents the shell,
and it interprets any argument as a file to take its input from, it runs the date command.
The file foo is known as a shell script. If we place some more command strings in this
file, then they will all be executed in a batch. Chapter 13 discusses shell scripting in detail.
All of these commands will be examined in some detail in the forthcoming chapters,
so let’s log out of the system. You know the technique of doing that by now. Generally,
exit terminates most sessions:
$ exit
login:

Caution

1.7

Make sure that you log out after your work is complete. If you don’t, anybody can get a hold
of your terminal and continue working using your user-id. She may even remove your files!
The login: prompt signifies a terminated session, so don’t leave your place of work until you
see this prompt.

How It All Clicked
Until UNIX came on the scene, operating systems were designed with a particular
machine in mind. They were invariably written in a low-level language (like assembler,
which uses humanly unreadable code). The systems were fast but were restricted to the

das76205_Ch01_001-024.indd 14

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

15

hardware they were designed for. Programs designed for one system simply wouldn’t
run on another. That was the status of the computer industry when Ken Thompson and
Dennis Ritchie, of AT&T fame, authored the UNIX system for their own use.
In 1969, AT&T withdrew its team from the MULTICS project, which was engaged
in the development of a flexible operating system that would run continuously and be
used remotely. Thompson and Ritchie then designed and built a small system with an
elegant file system, a command interpreter (the shell), and a set of utilities. To make
UNIX portable, they rewrote the entire system in the C language that was invented by
Ritchie himself. C is a high-level language, and programs coded in C run on all hardware.
For some time, portability remained one of the strong features of UNIX.
1.7.1

Berkeley: The Second School
A U.S. government law (subsequently revoked) prevented AT&T from selling computer
software. The company had no option but to distribute the product to academic and
research institutions at a nominal fee, but it came without any support. From the AT&T
product, the University of California, Berkeley (UCB), created a UNIX of its own. They
called it BSD UNIX (BSD—Berkeley Software Distribution). Both of these versions
became quite popular worldwide, especially in universities and engineering circles.
Later, UCB gave up all development work on UNIX.
Berkeley filled in the gaps left behind by AT&T, and then later decided to rewrite
the whole operating system in the way they wanted. They created the standard editor of
the UNIX system (vi) and a popular shell (C shell). Berkeley also created a better file
system, a more versatile mail feature, and a better method of linking files (symbolic
links). Later, they also offered with their standard distribution a networking protocol
software (TCP/IP) that powered the Internet. Like AT&T, they also offered it practically
free to many companies.

1.7.2

UNIX Gets Fragmented
Even though UNIX was written by programmers for programmers, its inherent strengths
found favor within business circles. Sun used the BSD System as a foundation for developing their own brand of UNIX (then SunOS). Today, their version of UNIX is known as
Solaris. Others had their own brands: IBM had AIX, HP offered HP-UX, while DEC produced Digital UNIX—and now Tru64 UNIX. Not to be left out, Apple offers Mac OS X, its
own open-source version of UNIX. The Linux wave arrived in the 90s, and most of these
vendors began to offer Linux too. Today, most supercomputers run UNIX, and handheld
devices increasingly use Linux, Google’s Android operating system being a case in point.
As each vendor modified and enhanced UNIX to create its own version, the original
UNIX lost its identity as a separate product. The BSD releases were much different from
the AT&T System V releases, and the incompatibilities steadily mounted. Finally, AT&T
took it upon themselves to unify many of these flavors into its last release—System V
Release 4 (SVR4). Shortly thereafter, AT&T sold its UNIX business to Novell, who
later turned over the UNIX trademark to a standards body called X/OPEN, now merged
with The Open Group.
The UNIX trademark is currently owned by The Open Group.

Note

das76205_Ch01_001-024.indd 15

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

16
1.7.3

The Internet
Even before the advent of SVR4, big things were happening in the U.S. Defense
Department. DARPA, a wing of the department, engaged several vendors to develop a
reliable communication system using computer technology. Through some brilliant work
done by Vinton Cerf and Robert Kahn, DARPA’s ARPANET network was made to work
using packet-switching technology. In this technology, data is split into packets, which
can take different routes and yet be reassembled in the right order. This was the birth of
TCP/IP—a set of protocols (rules) used by the Internet for communication.
DARPA commissioned UCB to implement TCP/IP on BSD UNIX. ARPANET
converted to TCP/IP in 1983, and in the same year, Berkeley released the first version
of UNIX that had TCP/IP built-in. The computer science research community were all
using BSD UNIX, and the network expanded like wildfire. The incorporation of TCP/IP
into UNIX and its use as the basis of development were two key factors in the rapid
growth of the Internet (and UNIX).

1.7.4

The Windows Threat
In the meantime, however, Microsoft was doing great things with Windows—a graphical
user interface (GUI) that uses the mouse rather than arcane and complex command
options to execute a job. Options could be selected from drop-down menu boxes and
radio buttons, which made handling some of the basic operating system functions easier.
Windows first swept the desktop market (with Windows 3.1/95/98) and then made significant inroads into the server market (with Windows NT/2000) which had been long
dominated by UNIX.
When UNIX badly needed a Windows-type interface for its survival, the
Massachusetts Institute of Technology (MIT) introduced the X Window System—the
first windowing system for UNIX. X Window has many of the important features of
Microsoft Windows plus a lot more. Every flavor of UNIX now has X along with a host
of other tools that can not only handle files and directories but also update the system’s
configuration files.
When all is said and done, the power of UNIX derives from its commands and their multiple
options. No GUI tool can ever replace the find command that uses elaborate file-attribute
matching schemes to locate files.

Note

1.8

POSIX and the Single UNIX Specification
Even though Dennis Ritchie rewrote UNIX in C to make it portable, that didn’t quite
happen. UNIX fragmentation and the absence of a single conforming standard adversely
affected the development of portable applications. To address the issue, AT&T created
the System V Interface Definition (SVID). Later, X/Open (now The Open Group), a
consortium of vendors and users, created the X/Open Portability Guide (XPG). Products
conforming to this specification were branded UNIX95, UNIX98, or UNIX03 depending
on the version of the specification.
Still another group of standards, the Portable Operating System Interface for
Computer Environments (POSIX), was developed at the behest of the Institution of

das76205_Ch01_001-024.indd 16

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

17

Electrical and Electronics Engineers (IEEE). POSIX refers to operating systems in
general, but was based on UNIX. Two of the most-cited standards from the POSIX family are known as POSIX.1 and POSIX.2. POSIX.1 specifies the C application program
interface—the system calls. POSIX.2 deals with the shell and utilities.
In 2001, a joint initiative of X/Open and IEEE resulted in the unification of the two
standards. This is the Single UNIX Specification, Version 3 (SUSV3) that is also known
as IEEE 1003.1:2001 (POSIX.1). The “write once, adopt everywhere” approach to this
development means that once software has been developed on any POSIX-compliant
UNIX system, it can be easily ported to another POSIX-compliant UNIX machine with
minimum modifications. We make references to POSIX throughout this text, but these
references should be interpreted to mean the SUSV3 as well.
The Single UNIX Specification, Version 3 is available at http://www.unix.org/unix03.html. You
must frequently consult this document when you use a command, an option, or a system call
to confirm whether the usage is mandated by the specification.

Tip

1.9

Linux and GNU
Although UNIX finally turned commercial, Richard Stallman and Linus Torvalds had
different ideas. Torvalds is the father of Linux, the free UNIX that has taken the computer world by storm. Stallman runs the Free Software Foundation (formerly known as
GNU—a recursive acronym that stands for “GNU’s Not Unix!”). Many of the important
Linux tools were written and supplied free by GNU.
Linux is distributed under the GNU General Public License, which makes it mandatory for developers and sellers to make the source code public. Linux is particularly
strong in networking and Internet features, and is an extremely cost-effective solution in
setting up an Internet server or a local internet. Today, development on Linux is carried
out at several locations across the globe at the behest of the Free Software Foundation.
The most popular GNU/Linux flavors include Ubuntu, Fedora (formerly Red Hat),
SuSE, Debian, and Mandriva. These distributions include a plethora of software—from
C and C++ compilers to Java, interpreters like perl, php, python and tcl; browsers like
Firefox, Internet servers, and multimedia software. Much of the software can also be
downloaded free from the Internet. All the major computer vendors (barring Microsoft)
have committed to support Linux, and many of them have ported their software to this
platform. This book also discusses Linux.

1.10

The UNIX Architecture
The entire UNIX system is supported by a handful of essentially simple, though somewhat
abstract concepts. The success of UNIX, according to Thompson and Ritchie, “lies not
so much in new inventions but rather in the full exploitation of a carefully selected set
of fertile ideas, and especially in showing that they can be keys to the implementation
of a small and yet powerful operating system.” UNIX is no longer a small system, but
it certainly is a powerful one. Before we examine the features of UNIX, we need to
understand its architecture—its foundation.

das76205_Ch01_001-024.indd 17

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

18

Division of Labor: Kernel and Shell
Foremost among these “fertile ideas” is the division of labor between two agencies—the
kernel and the shell. The kernel interacts with the machine’s hardware, and the shell
with the user. You have seen both of them in action in the hands-on session, though the
kernel wasn’t mentioned by name. Their relationship is depicted in Fig. 1.1.
The kernel is the core of the operating system. The system’s bootstrap program
(a small piece of program code) loads the kernel into memory at startup. The kernel
comprises a set of routines mostly written in C that communicate with the hardware
directly. User programs (the applications) that need to access the hardware (like the hard
disk or the terminal) communicate with the kernel using a set of functions called system
calls, which we discuss shortly.
Apart from providing support to user programs, the kernel has a great deal of
housekeeping to do. It manages the system’s memory, schedules processes, decides
their priorities, and performs other tasks which you wouldn’t want to bother about. The
kernel has work to do even if no user program is running. It is often called the operating
system—a program’s gateway to the computer’s resources.
Computers don’t have any inherent ability to translate user commands into action.
That requires an interpreter, and that job in UNIX is handled by the “outer part” of the
FIGURE 1.1

The Kernel-Shell Relationship

User

Shell

Shell

User

Shell

oft

wa

re

User

he

rS

cc

Ot

cp

ls

sort

Hardware

grep

Sp

sed

Data

base

das76205_Ch01_001-024.indd 18

ow
s

who

s

et

ps
s

Br

he

s
ad

er

re

tar

Other Compilers

X Window

Kernel

s

1.10.1

ssors

roce

P
Text

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

19

operating system—the shell. It is actually the interface between the user and the kernel.
Even though there’s only one kernel running on the system, there could be several shells
in action—one for each user who is logged in.
When you enter a command through the keyboard, the shell thoroughly examines
the keyboard input for special characters. If it finds any, it rebuilds a simplified command
line, and finally communicates with the kernel to see that the command is executed. This
interpretive action of the shell is examined in detail in Chapter 6.

Note

UNIX fragmentation becomes quite evident when you attempt to locate the kernel on your
system. It is often named unix (genunix on Solaris) and could be located in directories /unix
or /kernel. The shells are all available in /bin or /usr/bin.

The kernel is represented by the file /boot/vmlinuz. The shells are in /bin and
/usr/bin.
Linux

1.10.2

The File and Process
Two simple entities support the UNIX system—the file and process—and Kaare Christian
(The UNIX Operating System, John Wiley) makes two powerful abstractions about
them: “Files have places and processes have life.” Files are containers for storing static
information. Even directories and devices are considered files. A file is related to another
file by being part of a single hierarchical structure called the file system. Further, using
the cd and pwd (1.6.4) commands, you can “place” yourself at a specific location in this
hierarchy. Chapters 3 and 4 discuss file and directory handling.
The second entity is the process, which represents a program in execution. Like
files, processes also form a hierarchy, and are best understood when we consider one
process as the child of another. Unlike files, which are static entities, processes resemble
living organisms that are born and die. UNIX provides the tools that allow us to control
processes, move them between foreground and background, and even kill them. The
basics of the process management system are discussed in Chapter 7.

1.10.3

The System Calls
The UNIX system—comprising the kernel, shell, and applications—is written in C.
Though there are over a thousand different commands in the system, they often need
to carry out certain common tasks—like reading from or writing to disk. The code for
performing disk I/O operations is not built into the programs but is available in the kernel. Programs access these kernel services by invoking special functions called system
calls. Often the same system call can access both a file and a device; the open system
call opens both.
C programmers on a Windows system use the standard library functions for everything. You can’t use the write system call on a Windows system; you need to use
a library function like fprintf for that purpose. In contrast, the C programmer in the
UNIX environment has complete access to the entire system call library as well as the
standard library functions. You can use both write and fprintf in a C program meant
for running on a UNIX system.

das76205_Ch01_001-024.indd 19

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

20

POSIX specifies the system calls that all UNIX systems must implement. Once
software has been developed on one UNIX system using the calls mandated by POSIX,
it can be easily moved to another UNIX machine. Chapters 17 and 18 deal with the basic
system calls that you need to know to program in the UNIX environment.

1.11

Features of UNIX
UNIX is an operating system, so it has all of the features an operating system is expected
to have. However, UNIX also looks at a few things differently and possesses features
unique to itself. The following sections present the major features of this operating system.

1.11.1

Note

1.11.2

Note

das76205_Ch01_001-024.indd 20

A Multiuser System
From a fundamental point of view, UNIX is a multiprogramming system. It permits
multiple programs to remain in memory and compete for the attention of the CPU. These
programs can be run by different users; UNIX is also a multiuser system. This feature
often baffles Windows users as Windows is essentially a single-user system where the
CPU, memory, and hard disk are all dedicated to a single user. The who output (1.6.1)
showed three users working on the system.
For cycling through multiple jobs, the kernel uses the principle of time-sharing.
It breaks up a unit of time into several slices, and a user’s job runs for the duration of a
slice. The moment the allocated time expires, the previous job is kept in abeyance and
the next job is taken up. This process goes on until the clock has turned full circle and
the first job is taken up once again. This switching happens several times in one second,
so every user has the feeling that the machine is completely dedicated to her.
A program can leave the CPU before its time quantum expires if it performs an operation that
keeps the CPU idle. This has already been discussed in Section 1.1.

A Multitasking System Too
A single user can also run multiple tasks concurrently; UNIX is a multitasking system.
It is common for a user to edit a file, print another one on the printer, send email to a
friend, and browse the World Wide Web—all without leaving any of the applications.
The X Window system exploits the multitasking feature by allowing you to open multiple
windows on your desktop.
In a multitasking environment, a user sees one job running in the foreground; the
rest run in the background. You can switch jobs between background and foreground,
suspend, or even terminate them. As a programmer you can use this feature in a very
productive way. You can edit a C program with the vi editor and then suspend the vi
process to run the gcc compiler. You don’t need to quit vi to do that. This feature is
provided by most shells.
Today, we have machines with multiple CPUs that make it possible to actually earmark an entire
processor for a single program (in a single-user and single-tasking situation).

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

21

1.11.3

A Repository of Applications
By one definition, UNIX represents the kernel, but the kernel by itself can’t do anything
that can benefit the user. To exploit the power of UNIX, you need to use the host of
applications that are shipped with every UNIX system. These applications are quite
diverse in scope. There are general-purpose tools, text manipulation utilities (called
filters), compilers and interpreters, networked applications, and system administration
tools. You’ll also have a choice of shells.
This is one area that’s constantly changing with every UNIX release. New tools
are being added, and the older ones are being removed or modified. The shell and an
essential subset of these applications form part of the POSIX specification. There are
open-source versions for most of these utilities, and after you have read Chapter 16,
you should be able to download these tools and configure them to run on your machine.

1.11.4

The Building-Block Approach
One of the strengths of UNIX emerges from the designers’ belief that “small is beautiful.”
A complex task can be broken into a finite number of simple ones. The shell offers a
mechanism called the pipe that allows the output of one command to serve as input to
another. To take advantage of this feature a special set of commands (called filters) were
designed where each command did “one thing well.” By interconnecting these tools
using the piping mechanism, you can solve very complex text manipulation problems.
You can now understand why the who output (1.6.1) doesn’t display a header. If we
wanted to count the number of users by connecting the who output to a word-counting
program (like wc), a header line would have resulted in an erroneous count. This approach
also explains why most commands are not interactive. If a command had to pause to
take user input then it can’t be scheduled to run at a certain time of the day. Its output
can’t be used by another program without user intervention.

1.11.5

Pattern Matching
UNIX features very sophisticated pattern matching features. Many commands use filenames as arguments, and these filenames often have a common string. For instance, all
C programs have the .c extension, and to back them up to tape with the tar command,
we need not specify all of their filenames to tar. Instead, we can simply use a pattern
*.c. The * is a special character (known as a metacharacter) that is used by the shell
to match a number of characters. If you choose your filenames carefully, you can use a
simple expression to access a whole lot of them.
Pattern matching isn’t confined to filenames only. Some advanced tools (like grep,
sed and awk) also use a different metacharacter set for matching strings contained in files.
In this scheme, a pattern represented by printf.*name matches all lines that contain
both printf and name. This pattern is called a regular expression. This book heavily
emphasizes the importance of regular expressions and shows how you can perform
complex pattern matching tasks using them.

1.11.6

Programming Facility
The UNIX shell is also a programming language; it was designed for a programmer, not
a casual end user. It has all the necessary ingredients, like control structures, loops, and

das76205_Ch01_001-024.indd 21

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

22

variables, that establish it as a powerful programming language in its own right. These
features are used to design shell scripts—programs that run UNIX commands in a batch.
Many of the system’s functions can be controlled and automated by using these
shell scripts. If you intend to take up system administration as a career, then you’ll have
to know the shell’s programming features very well. Proficient UNIX programmers seldom refer to any other language (except Perl and PHP) for text manipulation problems.
Shell programming is taken up in Chapter 13.
1.11.7

Documentation
UNIX documentation is no longer the sore point it once was. Even though it’s sometimes
uneven, usually the treatment is quite lucid. The principal online help facility available
is the man command, which remains the most important reference for commands and
their configuration files. Today there’s no feature of UNIX on which a separate textbook
is not available. UNIX documentation and the man facility are discussed in Chapter 2.
Apart from the online documentation, there’s a vast ocean of UNIX resources
available on the Internet. There are several newsgroups on UNIX where you can post
your queries in case you are stranded with a problem. The FAQ (Frequently Asked
Questions)—a document that addresses common problems—is also widely available
on the Net. Then there are numerous articles published in magazines and journals and
lecture notes made available by universities on their Web sites.
With the goal of building a comfortable relationship with the machine, Thomson
and Ritchie designed a system for their own use rather than for others. They could afford to do this because UNIX wasn’t initially developed as a commercial product, and
the project didn’t have any predefined objective. They acknowledge this fact too: “We
have not been faced with the need to satisfy someone else’s requirements, and for this
freedom we are grateful.”

SUMMARY
A computer needs an operating system (OS) to allocate memory, schedule programs,
and control devices. The UNIX system also provides a host of applications for the use
of programmers and users.
Multiprogramming systems like UNIX allow multiple programs to reside in
memory. Even though a program may run for the duration of the time slice allocated
for it, it may prematurely leave the CPU during a blocking operation (like reading a
file) that keeps the CPU idle.
You enter a UNIX system by entering a user-id and a password. You can terminate
a session by using the exit or logout command or pressing [Ctrl-d].
UNIX commands are generally in lowercase. date displays the system date and
time. who displays the list of users logged on to the system. ps lists all processes running
at a terminal. It always shows the shell process running.
You can display a file with cat, copy it with cp, rename it with mv, and remove
it with rm.
mkdir creates a directory, pwd displays the pathname of the current directory, and
cd changes the current directory. rmdir removes an empty directory.

das76205_Ch01_001-024.indd 22

12/13/11 10:19 AM

Chapter 1: Introducing UNIX

23

UNIX was developed at AT&T Bell Laboratories by Ken Thompson and Dennis
Ritchie. It was finally written in C. Notable work was also done at Berkeley. AT&T
introduced System V Release 4 (SVR4) to merge their own version, Berkeley, and other
variants.
Linux is a UNIX implementation that is constantly growing with contributions
from the Free Software Foundation (formerly GNU).
Modifications to the system made by vendors led to both enhancement and fragmentation of UNIX. Two merged standards, POSIX and the Single UNIX Specification,
are today used as guidance for development work on UNIX.
All work is shared by the kernel and shell. The kernel manages the hardware, and
the shell interacts with the user. The shell and applications communicate with the kernel
using system calls, which are special routines built into the kernel.
The file and process are the two basic entities that support the UNIX system. UNIX
considers everything to be a file. A process represents a program (a file) in execution.
UNIX is a multiuser and multitasking system. Several users can use the system
together, and a single user can also run multiple jobs concurrently.
UNIX uses a building-block approach in the design of some of its tools and lets
you develop complex command routines by connecting these tools.
The UNIX man command is the primary online help facility available.

SELF-TEST
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
1.10
1.11
1.12
1.13
1.14
1.15

das76205_Ch01_001-024.indd 23

The ________ interacts with the hardware and the _________ interacts with the
user.
A program is synonymous with a process. True or false?
Every character has a number associated with it. What is it called?
If you see a prompt like mailhost login:, what do you think mailhost
represents?
If the system echoes Login incorrect, does it mean that your user-id is
incorrect?
Name the commands you used in this chapter to display (i) filenames,
(ii) processes, (iii) users.
Run ps and note the PID of your shell. Log out and log in again and run ps
again. What do you observe?
Create two files, foo1 and foo2, with the echo command, and then use
cat foo1 foo2. What do you observe?
Now run the command cat foo[12], and note your observations.
Enter the command echo SHELL. What mistake did you make?
Create a file foo containing the words hello dolly. Now create a directory bar,
and then run mv foo bar. What do you observe when you run both ls and ls bar?
Who are the principal architects of the UNIX operating system?
Why did AT&T virtually give away UNIX to the world?
Where did BSD UNIX originate? Name some features of UNIX that were first
found in BSD UNIX.
Which flavor of UNIX is available for free and runs on the PC?

12/13/11 10:19 AM

Your UNIX/Linux: The Ultimate Guide

24
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25

Identify the companies associated with the following brands: (i) Solaris, (ii) AIX,
(iii) Tru64 UNIX.
What does X/OPEN represent? Who owns the UNIX trademark today?
Who are the two brains behind Linux?
What is the distinctive characteristic about the GNU General Public License?
Why is UNIX more portable than other operating systems?
Can you divide UNIX into two major schools? To which school does Sun’s UNIX
belong?
Why do UNIX tools perform simple jobs rather than complex ones?
What is the windowing system of UNIX known as?
Name some interpretive languages available on UNIX systems.
Name three notable Linux flavors.

EXERCISES
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9

Operating systems like UNIX provide services both for programs and users. Explain.
What does a program do when it needs to read a file?
Does a program always complete its time quantum before it makes way for
another program?
Explain the significance of the terms multiprogramming, multiuser, and
multitasking.
Why are UNIX commands noninteractive, and why is their output not usually
preceded by header information?
What are system calls, and what role do they play in the system? How is C programming so different and powerful in the UNIX environment compared to Windows?
Two UNIX systems may not use the same system calls. True or false?
Name the three commands that you would try in sequence to log yourself out of
the system. Which one of them will always work?
Run the following commands, and then invoke ls. What do you conclude?
echo > README [Enter]
echo > readme [Enter]

1.10
1.11
1.12
1.13
1.14

1.15
1.16

das76205_Ch01_001-024.indd 24

Enter the following commands, and note your observations: (i) who and tty,
(ii) tput clear, (iii) id, (iv) ps and echo $$.
When you log in, a program starts executing at your terminal. What is this program known as? Name four types of this program that are available on a system.
What is the significance of your user-id? Where in the system is the name used?
What are the two schools of UNIX that initially guided its development? Mention
the outcome of the standardization efforts that are currently in force today.
Create a directory, and change to that directory. Next, create another directory
in the new directory, and then change to that directory too. Now, run cd without
any arguments followed by pwd. What do you conclude?
Why is the shell called a command interpreter?
What is the one thing that is common to directories, devices, terminals, and
printers?

12/13/11 10:19 AM

CHAPTER

2

Becoming Familiar
with UNIX Commands

A

major part of the job of learning UNIX is to master the essential command set.
UNIX has a vast repertoire of commands that can solve many tasks either by
working singly or in combination. In this chapter, we’ll examine the generalized UNIX
command syntax and come to understand the significance of its options and arguments.
The complete picture of command usage is available in the man pages, and we’ll learn
to look up this documentation with the man command.
We’ll next try out some of the general-purpose utilities of the system. We’ll change
the password and get comfortable with email using a command-line tool. We’ll learn
about other tools that tell us the date, the users of the system, and some specifics of
the operating system. At times we need to consider situations where the output of these
commands can be processed further. Finally, we take a look at the common traps that
befall the user and how the stty command can change many keyboard settings.

Objectives
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Understand the breakup of the command line into arguments and options.
Learn how the shell uses the PATH variable to locate commands.
Learn how commands can be used singly or in combination.
Use the man command to browse the UNIX documentation.
Understand the organization of the documentation.
Display messages with echo, and understand why printf is superior.
Save all keystrokes and command output in a file with script.
Understand email basics and why you need a command-line email program like mailx.
Use passwd to change your own password.
Know your machine’s name and operating system with uname.
Find out the users of the system with who.
Display the system date in various formats with date.
Know what can go wrong, and use stty to change keyboard settings.
Get introduced to the X Window system.

25

das76205_Ch02_025-055.indd 25

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

26

2.1 Command Basics
UNIX commands are generally implemented as disk files representing executable
programs. They are mainly written in C, but UNIX supports programs written in any
language. When you run a command, the program is loaded into memory, and the CPU
starts executing the instructions contained in the program.
UNIX is sensitive to the case of filenames, though command names are generally in
lowercase. These names are seldom more than four characters long. You can sometimes
deduce the function from the name (like cp for copy) but sometimes not (like grep for
searching for a pattern).
Unlike Windows, UNIX doesn’t require command names to have an extension
(like .exe, .com, etc.). Extensions are used either for convenience or for conforming to
a requirement imposed by the application. For instance, C and Java programs need to
have the .c and .java extensions, respectively, because their respective compilers won’t
work otherwise. However, shell and Perl scripts don’t need the .sh or .pl extensions,
though we often provide them for easier identification.
There’s one command that’s special—the shell. Unlike other commands, the
shell is invoked the moment you log in and continues to run until you log out. We’ll be
constantly making references to the shell’s behavior in these initial chapters before we
embark on a detailed examination in Chapter 6.

Note

2.1.1

In case you begin to feel that all commands are loaded from disk, we need to make the distinction between external and internal commands. External commands exist on disk as separate files.
But there’s also a separate set of commands that are built into the shell executable. You have
used one of them already—the cd command (1.6.4). There’s no file in the system named cd.

The PATH: Locating Commands
How does the shell know whether a command can be executed or not? The shell maintains a variable named PATH in its own environment (thus also known as an environment
variable). PATH is set to a list of colon-delimited directories. You can display this list by
simply evaluating the $-prefixed variable with the echo command:
$ echo $PATH
/bin:/usr/bin:/usr/ucb:/usr/xpg4/bin:.

Output in the C shell
is a little different

There are five directories in this list separated by a delimiter, the colon. A delimiter
separates one piece of information from another, a feature exploited by many UNIX
files (like the one that stores your password). The fifth directory is signified by a dot
that represents the current directory. When you enter a command, the shell looks in each
of these directories to locate the file with that name. The following message shows that
the netscape command is not available in any of these directories:
$ netscape
bash: netscape: not found

das76205_Ch02_025-055.indd 26

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

27

The Bash shell is running here, and it prints the message after failing to locate the file.
This doesn’t in any way confirm that netscape doesn’t exist on this system; it could
reside in a different directory. In that case we can still run it by
• changing the value of PATH to include that directory.
• using a pathname (like /usr/local/bin/netscape if the command is located in
/usr/local/bin).
• switching to the directory where the executable is located and executing it from
there.
Windows users also use the same PATH variable to specify the search path, except
that Windows uses the ; as the delimiter instead of the colon. Windows also uses the
dot to signify the current directory. We have more to say about pathnames in Chapter 3
and we’ll learn to change PATH in Chapter 8.
Is it a good idea to have the current directory at the end of PATH? Yes, this location provides a safeguard against inadvertent execution of programs that have the same
name as the ones resident in /bin and /usr/bin. This means that if you have mistakenly
provided the name cat to one of your programs, invoking cat won’t execute the one you
have created and placed in the current directory. But what if you still want to execute your
own cat program? Simply invoke it as ./cat, in which case, PATH would be ignored.

Note

2.1.2

The essential UNIX commands for general use are located in the directories /bin and /usr/bin.
The commands showing graphical output are usually found in /usr/X11R6/bin or /usr/dt/bin.
The commands used by the system administrator are found in /sbin and /usr/sbin.

Where Is the Command?
There are three commands that provide clues to the location of another command—which,
whereis and type. Sometimes you’ll want to know whether a command is available in
PATH before you decide to execute it. The which command tells you the directory that
contains the command:
$ which grep
/usr/bin/grep

After searching the directories of PATH in sequence, which abandons its search the moment it locates a file named grep. This is the global regular expression printer—a
command that searches for patterns in a file. You may not have which on your system, and
POSIX doesn’t require UNIX systems to offer this utility. If you are using a BSD-based
UNIX system, then you can try the whereis command. This time let’s try to locate ls:
$ whereis ls
ls: /usr/bin/ls /usr/ucb/ls

Berkeley version also shown

Unlike which, which confines its search to the directories in PATH, whereis looks up a
larger list. It finds two versions of ls on this Solaris system. Note that whereis is not
also supported by POSIX.

das76205_Ch02_025-055.indd 27

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

28

As noted in a previous aside, not all UNIX commands are files; some are built
into the shell. which and whereis don’t consider this possibility, so if you look for the
echo command, which says that it is in /usr/bin:
$ which echo
/usr/bin/echo

This is not the file that is executed when we run echo; modern shells have echo
built in. The information output by which is only half true; a more authentic picture is
presented by the type command:
$ type echo
echo is a shell builtin

So even if which shows the pathname of echo, type makes it clear that the shell will
always run its own built-in command. DOS users are reminded that type doesn’t display
files but only points to the version that will be executed.
which locates a command’s disk file, but ambiguity arises when the command is also a shell
builtin. As a general rule, the shell will ignore the disk version in preference to its own builtin.
(However, we can override this preferential behavior.) The type command provides a more
reliable picture of the command that is actually executed when it exists in different forms. type
itself is a shell builtin and is not available in some versions of the C shell.

Note

2.2 Command Structure
In Chapter 1, we used commands that had multiple words (like mkdir scripts) and
one that had an embedded minus sign (ls -F). It’s time we subjected a typical UNIX
command to a dissective study. The structure of such a command is shown in Fig. 2.1.
This command sequence has five words. The first word is the command itself
and the remaining ones are its arguments. The ls command is specified here with four
arguments. Two of the arguments begin with a hyphen (-l and -t) and are appropriately
called options. The entire line is referred to as the command line. A command line is
executed only after you hit [Enter].
Every command has a fixed set of options. An option changes a command’s default
behavior, so if ls shows only the filenames, the -l and -t options show their attributes
as well. Some commands use files (note1 and note2 here) and some don’t. If used at
FIGURE 2.1

Structure of a UNIX Command
Options
ls
Command

das76205_Ch02_025-055.indd 28

-l

-t

note1

note2

Arguments

12/13/11 10:43 AM

29

Chapter 2: Becoming Familiar with UNIX Commands

all, the filename will generally be a command’s last argument—and after all options.
(This is not always true; some options use filenames as their own “arguments.”)

Caution

Beginners often forget to provide spaces between the command and the argument. You can
get away with DIR/P instead of DIR /P in the DOS environment of Windows, but in UNIX you
need to be careful:

$ ls-w
bash: ls-w: command not found
The shell fails to recognize -w as an argument and treats ls-w as a single command. Obviously,
this command doesn’t exist in PATH.

A command is separated from its options and arguments by whitespace. This is
a collective term that comprises a contiguous string of spaces and tabs (and newlines).
You can liberally provide a lot of whitespace in the command line, but the shell ensures
that arguments are separated by a single space before the command is run.
Not all error messages are generated by the shell. When you use a command with
an incorrect option, the shell locates the command all right, but the command this time
finds the option to be wrong:
$ ls -z note
/bin/ls: invalid option -- z
usage: ls -1RaAdCxmnlogrtucpFbqisfL [files]

Error message from ls

ls does have a large number of options (over 20), but it seems that -z is not one of
them. Many commands also display the correct syntax and options when you use them
incorrectly.

Caution

Linux

Never create a filename that begins with a hyphen; many commands just won’t work! It’s
common for a command to treat a filename beginning with a hyphen as one of its options.
This often results in erroneous output. The previous ls command would have reacted no differently even if there was a file named -z. If you create or find such a file in your work area,
you can still list it with ls ./-z and remove it with rm ./-z. You’ll soon learn the convenience
of the ./ prefix in explicitly specifying a file in the current directory. We don’t normally use
cat ./foo (cat foo is good enough), but when handling filenames like -z, the ./ prefix can
be of great help.

Linux offers all of the UNIX-type options, but it also offers options using two hyphens
and a meaningful word. For instance, it offers the synonym ls --classify in addition to ls -F. The expanded options are easy to remember; for example, it’s easier to
remember --classify than -F.
Options can often be combined with only one - sign. Take for instance this command line containing three options:
ls -l -a -t

das76205_Ch02_025-055.indd 29

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

30

ls is taken up in detail in Chapter 4, so don’t bother about the significance of these
three options. These are simple options, and UNIX lets you combine them in any order:
ls -lat
ls -tal

Sequence of combination not always
important, but sometimes is

Don’t interpret the preceding discussions as a general prescription. Some commands
won’t let you combine options, as shown above, and some have options that begin with
a +. Moreover, awk and perl use a programming script as an argument. But don’t let
this deter you; you will have already built up a lot of muscle before you take on these
commands.
C programmers and shell scripters need to count the number of arguments in their programs.
It helps to be aware at this stage that there are some characters in the command line that
are not really arguments—the |, > and <, for instance. In Chapter 6, we’ll make an amazing
discovery that in the command line who > foo, foo is not an argument to who!

Note

2.3

Flexibility of Command Usage
So far we have executed commands in sequence—by waiting for the prompt to return
before keying in the next command. UNIX provides a certain degree of flexibility in
the usage of commands. We can enter multiple commands in one line, and their output
need not always come to the terminal. The following discussions actually belong to the
domain of the shell, but we need to be aware of them even if we don’t understand fully
how they work:
You don’t have to wait for a command to complete before you type your next
command. UNIX provides a full-duplex terminal, which provides separate channels for
input and output. Just go on typing even if the output of the previous command clutters
the display. All commands are stored in a keyboard buffer (a temporary form of storage)
and will eventually be passed on to the shell.
You can specify multiple commands in one line. Using a ; as the delimiter of
commands, you can specify more than one command before you hit [Enter]:
who ; ls note

ls executed after who

Here we have two command lines on a single line. The ; is gobbled up by the shell and
is not seen by the who command. The ; is a metacharacter, and the UNIX shell understands a large number of metacharacters. You’ll encounter many of them in Chapter 6.
A command line can be split into multiple physical lines. Sometimes you’ll find it
either convenient or necessary to spread out the command line into multiple lines. This
is how the echo command works with most shells:
$ echo “This is[Enter]
> a three-line[Enter]
> text message”[Enter]
This is
a three-line
text message

das76205_Ch02_025-055.indd 30

A second prompt (>) appears . . .
. . . and disappears after quote is closed

12/13/11 10:43 AM

31

Chapter 2: Becoming Familiar with UNIX Commands

Here we hit [Enter] twice to see the secondary prompt (>) appear, which disappeared
after we closed the quote. The appearance of the > (or ?) indicates that the command
line isn’t complete.

C Shell

The preceding echo command won’t work with the C shell. You’ll have to enter a
\ (backslash) before you press [Enter]. Moreover, the C shell often throws out a
different secondary prompt (?). Its primary prompt is also different (%):
% echo “This is\[Enter]
? a three-line\[Enter]
? text message”

Tip

Some C shells don’t show
the ? at all.

Whenever you find the > or ? appearing after you have pressed [Enter], it will often be due to
the absence of a matching quote or parenthesis. In case you find that the problem persists even
after providing it, just interrupt the command line with [Ctrl-c] or [Ctrl-u]. The significance of
these keys is taken up in Section 2.14.

Command output need not always be seen on the terminal. We often save output
in files like this:
who > userlist.txt

Output saved in userlist.txt

As discussed in Section 1.11.4, UNIX uses a modular approach to solving problems.
The output of one command can be useful input for another:
who | wc -l

who piped to wc -l

This sequence counts the number of users logged in. Chapter 6 discusses how the
> and | direct the shell to make the necessary I/O connections that could be used
by programs.

2.4 man: On-Line Help
UNIX is ultimately mastered by looking up its documentation, which is available today
in a number of forms of varying complexity. The earliest and most important is the one
that is viewed with the man command—often called the man documentation. man remains
the most complete and authoritative guide to the UNIX system. The documentation is
also available in print and on the Internet.
To view the manual page of the wc command, use man with wc as argument:
man wc

The entire man page is dumped onto the screen (Fig. 2.2). man presents the first page
and pauses. It does this by sending its output to a pager program, which displays the
contents of a file one page (screen) at a time. The pager is actually a UNIX command,

das76205_Ch02_025-055.indd 31

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

32
FIGURE 2.2

The man page for wc (Solaris)

User Commands

wc(1)

NAME
wc - display a count of lines, words and characters in a file
SYNOPSIS
wc [ -c | -m | -C ] [ -lw ] [ file ... ]
DESCRIPTION
The wc utility reads one or more input files and, by default,
writes the number of newline characters, words and bytes contained
in each input file to the standard output.
The utility also writes a total count for all named files, if more
than one input file is specified.
wc considers a word to be a non-zero-length string of characters
delimited by white space (for example, SPACE, TAB ). See
iswspace(3C) or isspace(3C).
OPTIONS
The following options are supported:
-c
Count bytes.
-m
Count characters.
-C
Same as -m.
-l
Count lines.
-w
Count words delimited by white space characters or new line
characters. Delimiting characters are Extended Unix Code (EUC)
characters from any code set defined by iswspace().
If no option is specified the default is -lwc (count lines, words,
and bytes.)
OPERANDS
The following operand is supported:
file A path name of an input file. If no file operands are
specified, the standard input will be used.
USAGE
See largefile(5) for the description of the behavior of wc when
encountering files greater than or equal to 2 Gbyte (2 **31 bytes).
EXIT STATUS
The following exit values are returned:
0
Successful completion.
>0
An error occurred.
SEE ALSO
cksum(1),
setlocale(3C),

das76205_Ch02_025-055.indd 32

isspace(3C),
attributes(5),

iswalpha(3C),
environ(5),

iswspace(3C),
largefile(5)

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

33

and man is always preconfigured to be used with a specific pager. UNIX systems currently use these pager programs:
• more, Berkeley’s pager, which is now available universally as a superior alternative to the original AT&T pg command (now obsolete). We’ll be considering more
in this text.
• less, the standard pager used on Linux systems, but also available for all UNIX
platforms. less is modeled on the vi editor and is more powerful than more because
it replicates many of vi’s navigational and search functions. The features of less
are described briefly in Section 3.16.
On a man page that uses more as the pager, you’ll see a prompt at the bottom-left
of the screen which looks something like this:
--More--(26%)

less shows a : as the prompt

At this prompt you can press a key to perform navigation or search for a string. The
key you press can be considered to be one of man’s (rather, the pager’s) internal commands, and the character represented by the key often doesn’t show up on the screen.
Many UNIX utilities like vi and mailx also have their own internal commands. A set
of internal commands used by more is listed in Table 3.3. We’ll discuss only a few of
them related to navigation and string search.
To quit the pager, and ultimately man, press q. You’ll be returned to the shell’s
prompt.
2.4.1

Navigation and Search
The navigation commands are numerous and often vary across UNIX implementations.
For the time being, you should know these two commands, which should work on all
systems:
f or spacebar—Advances by one screen.
b—Moves back one screen.
The man documentation is sometimes quite extensive, and the search facility lets you
locate a page containing a keyword quite easily. For example, you can call up the page
containing the word clobber by following the / (frontslash) with the term:
/clobber[Enter]

The / and search string show up on the screen this time, and when you press [Enter],
you are taken to the page containing clobber. If that’s not the page you are looking
for, you can repeat the search by pressing n. Some pager versions even highlight the
search term in reverse video.
2.4.2

das76205_Ch02_025-055.indd 33

Further Help with man -k and man -f
POSIX requires man to support only one option (-k). Most UNIX systems also offer the
apropos command that emulates man -k. When used with this option, man searches

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

34

the NAME section of all man pages that contain the keyword. To know more about the
cron facility, use
$ man -k cron
cron
cron (1m)
crontab
crontab (1)
queuedefs
queuedefs (4)

- clock daemon
- user crontab file
- queue description file for at, batch, and cron

cron is the UNIX scheduler that takes instructions from a crontab file, and to know
more about it (Chapter 7), you need to look up the man pages of cron and crontab.
Note that cron and crontab are documented in Sections 1m and 1, respectively, on
this Solaris system.
The -f option simply displays a one-line header from the NAME section. The
whatis command emulates man -f. This is what grep does:
$ man -f grep
grep
grep (1)

- search a file for a pattern

We use grep throughout this book for handling most pattern search issues. If this
is the command you need, you can use man grep to learn that a pattern can match
multiple strings.

2.5 The man Documentation
Vendors organize the man documentation differently, but in general you could see eight
sections of the UNIX manual (Table 2.1). Later enhancements have added subsections
(like 1C, 1M, 3N, etc.), but we’ll ignore them in this text. You can see from the table
that the documentation is not restricted to commands; important system files used by
these commands and system calls also have separate man pages.
Most of the commands discussed in this text are available in Section 1, and man
searches the manuals starting from Section 1. If it locates a command in one section, it
won’t continue the search even if the command also occurs in another section. When a
keyword is found in multiple sections, you should use the section number additionally
TA B L E 2 . 1

das76205_Ch02_025-055.indd 34

Organization of the man Documentation

Section

Subject (Solaris)

Subject (Linux)

1
2
3
4
5
6
7
8

User programs
Kernel’s system calls
Library functions
Administrative file formats
Miscellaneous
Games
Special files (in /dev)
Administration commands

User programs
Kernel’s system calls
Library functions
Special files (in /dev)
Administrative file formats
Games
Macro packages and conventions
Administration commands

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

35

as an argument. Depending on the UNIX flavor you are using, you may also need to
prefix the -s option to the section number:
man 4 passwd
man -s4 passwd

passwd also occurs in Section 4
Solaris uses the -s option

This displays the documentation for a configuration file named /etc/passwd from
Section 4. There’s also an entry for passwd in Section 1, but if we had used man passwd
(without the section number), man would have looked up Section 1 only and wouldn’t
have looked at Section 4 at all.
If you are using the X Window system, then you can use the xman graphic client
to view man pages. Simply execute the command in any terminal window.

Note

2.5.1

There are two chapters in this text that feature the important system calls and some standard
library functions. Sections 2 and 3 of the man documentation provide detailed accounts of
their usage. To look up the read system call, you’ll have to use man 2 read or man -s2 read.

Understanding a man Page
A man page is divided into a number of compulsory and optional sections. Not every
command has all sections, but the first three (NAME, SYNOPSIS and DESCRIPTION)
are generally seen in all man pages. NAME presents a one-line introduction to the command. SYNOPSIS shows the syntax used by the command, and DESCRIPTION (often
the largest section) provides a detailed description.
The SYNOPSIS section is the one that we need to examine closely, and we’ll do
that with reference to the man page of the wc command shown in Fig. 2.2. Here you’ll
find the syntax—the options and arguments used with the command. The SYNOPSIS
follows certain conventions and rules which every user must understand:
• If a command argument is enclosed in rectangular brackets, then it is optional;
otherwise, the argument is required. The wc man page shows all its arguments
enclosed in three such groups. This means that wc can be used without arguments.
• The ellipsis (a set of three dots) implies that there can be more instances of the
preceding word. The expression [ file ... ] signifies that wc can be used with
more than one filename as argument.
• If you find a | (pipe) character in any of these areas, it means that only one of the
options shown on either side of the pipe can be used. Here, only one of the options,
-c, -m and -C, can be used.
All options used by the command are listed in the OPTIONS section. Often,
difficult options are supported by suitable examples. There’s a separate section named
EXIT STATUS which lists possible error conditions and their numeric representation.
You need to understand the significance of these numbers when writing shell scripts and
C programs in order to determine the actual cause of termination of a program.

2.5.2

Using man to Understand man
Since man is also a UNIX command like ls or cat, you’ll probably first like to know
how man itself is used. Use the same command to view its own documentation:
man man

das76205_Ch02_025-055.indd 35

Viewing man pages with man

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

36

From this man page you’ll know that you can choose your pager too. The variable,
PAGER, controls the pager man uses, and if you set it to less, then man will use less as
its pager. This is how you set PAGER at the command prompt before you invoke man:
PAGER=less ; export PAGER
man wc

Set this shell variable and export it
before you run man

To evaluate the value of PAGER, use the command echo $PAGER. This setting is valid
only for the current session. In later chapters, you’ll understand the significance of the
export statement and also learn to make this setting permanent so that its assigned value
remains valid for all sessions.

Note

On some systems, echo $PAGER may not show you any value at all, in which case man is using
a default pager. Some systems set this variable in the file /etc/default/man instead.

There can be more headers, but we have covered the major ones. All said and done,
commands having one or two man pages are generally easy to use, but not the ones that
have tens of pages. man is more of a reference than a tutorial, and the manuals are good
reading material only after you have sufficiently mastered the system.

info and --help: Two Important Help Resources
Linux

Man pages are read in a linear manner (from beginning to end) and have obvious
limitations. Linux offers two additional help facilities—the info command and the
--help option. Most commands support the --help option, though not all commands
have info pages.
info info is GNU’s info reader for browsing Texinfo documentation. Invoke it with
a command (say, info grep) to see the documentation organized in nodes (Fig. 2.3).
Each node is marked with an asterisk at the beginning of the line. As in Web pages,
there are multiple levels here, and the deeper you descend the more detailed the treatment becomes.
Use the [Tab] key to move to a node and hit [Enter]. You’ll see the current page
replaced with another. info is a little difficult to use at first, but if you remember these
four navigation commands initially, you should feel fairly comfortable:
n
p
u
l

Visits the next node
Visits the previous node
Returns to the previous level where [Enter] was pressed
Moves to the previously visited node

Within a page, you can use the [PageUp] and [PageDown] (or spacebar) keys
in the normal way for paging. You should be careful about using other keys because
you may get stuck in the middle. In case that happens and these keys don’t take you
to the desired point, just quit with q and reenter info. When in doubt, press h to see
the complete list of key sequences.

das76205_Ch02_025-055.indd 36

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

37

--help Some commands have just too many options, and sometimes a quick lookup
facility is what you need. The --help option displays a compact listing of all options.
Here’s an extract from the find --help output that shows its options in a compact
manner:
tests (N can be +N or -N or N): -amin N -anewer FILE -atime N -cmin N
-cnewer FILE -ctime N -empty -false -fstype TYPE -gid N -group NAME
-ilname PATTERN -iname PATTERN -inum N -ipath PATTERN -iregex PATTERN
-links N -lname PATTERN -mmin N -mtime N -name PATTERN -newer FILE
-nouser -nogroup -path PATTERN -perm [+-]MODE -regex PATTERN
-size N[bckw] -true -type [bcdpfls] -uid N -used N -user NAME
actions: -exec COMMAND ; -fprint FILE -fprint0 FILE -fprintf FILE FORMAT
-ok COMMAND ; -print -print0 -printf FORMAT -prune -ls

A Linux command invariably offers far more options than its UNIX counterpart.
You’ll find this lookup facility quite useful when you know the usage of the options,
but can’t recollect the one you require.
In the remainder of this chapter, we’ll examine a few general-purpose utilities that
you need to be familiar with. Many of these utilities report on the state of the system
and form important ingredients in shell programming.
FIGURE 2.3

The info Page of grep

File: grep.info, Node: Top, Next: Introduction, Up: (dir)
Grep
****
`grep’ searches for lines matching a pattern.
This document was produced for version 2.5.1 of GNU `grep’.
* Menu:
*
*
*
*
*
*
*
*
*
*

Introduction.
Invoking `grep’; description of options.
Exit status returned by `grep’.
`grep’ programs.
Regular Expressions.
Examples.
Reporting Bugs.
License terms.
A menu with all the topics in this manual.
A menu with all `grep’ commands
and command-line options.
--zz-Info: (grep.info.gz)Top, 23 lines --Top-- Subfile: grep.info-1.gz---------Welcome to Info version 4.2. Type C-h for help, m for menu item.

das76205_Ch02_025-055.indd 37

Introduction::
Invoking::
Diagnostics::
Grep Programs::
Regular Expressions::
Usage::
Reporting Bugs::
Copying::
Concept Index::
Index::

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

38

2.6 echo: Displaying Messages
We used the echo command in Section 1.6.3 to save some data in a file. The command
is often used in shell scripts to display diagnostic messages on your terminal or to issue
prompts for taking user input:
$ echo “Filename not entered”
Filename not entered

Shell version of echo used here

echo is often used to evaluate shell variables. This is how you find out the shell you
are using:
$ echo $SHELL
/bin/bash

Variables are evaluated with $
The Bash shell

UNIX fragmentation comes to light when you attempt to use echo with escape sequences.
The AT&T version of echo supports escape sequences, but not BSD. An escape sequence
begins with a \ and is followed by a single character or a zero-prefixed number. For
instance, \c is an escape sequence. When it is placed at the end of a string that’s used
with echo, the cursor is placed at the end of the output string rather than on the next line:
$ echo “Enter filename: \c”
Enter filename: $ _

Prompt and cursor in same line

This is how we use echo in a shell script to accept input from the terminal. Like \c, there
are other escape sequences (Table 2.2). Here are two commonly used ones:
\t—A tab which pushes text to the right by eight character positions.
\n—A newline which creates the effect of pressing [Enter].
There’s another type of escape sequence that uses ASCII octal values (numbers that use
the base 8 contrasted with the standard decimal system, which uses 10). echo interprets
a number as octal when it is preceded by \0. For instance, [Ctrl-g] (the BELL character)
has the octal value 07. This is how you can use echo to sound a beep:
$ echo “\07”
..... Beep heard .....
TA B L E 2 . 2

das76205_Ch02_025-055.indd 38

\007 will also do

Escape Sequences Used by echo and printf

Escape Sequence

Significance

\a
\c
\f
\n
\r
\t
\\
\0n

Bell
No newline (cursor in same line)
Formfeed
Newline
Carriage return
Tab
Backslash
ASCII character represented by the octal value n, where n can’t
exceed 377 (decimal value 255)

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

39

This is our first encounter with octal values as command arguments. Later, we’ll see
that the tr, awk and perl commands also use octal values. For reasons that are covered
later, it helps to enclose the arguments within quotes.

Caution

echo escape sequences are a feature of System V. BSD doesn’t recognize them, but it supports
the -n option as an alternative to the \c sequence:

echo “Enter filename: \c”
echo -n “Enter filename: “

System V
BSD

Even though we don’t use the disk version of echo nowadays, the bad news is that the shells
also respond in different ways to these escape sequences. Rather than go into these details, a
word of caution from POSIX would be appropriate: use printf.

Bash, the standard shell used in Linux, as well as Mac OS X, interprets the escape sequences
properly only when echo is used with the -e option:
BASH

echo -e “Enter filename: \c”
We’ll be using these escape sequences extensively in this text, so if you are a Bash user (which
most Linux users are), you must commit this option to memory.

2.7 printf: Alternative to echo
Unless you have to maintain a lot of legacy code that uses echo, choose printf. Like
echo, it exists as an external command, but it’s only Bash that has printf built in.
printf also recognizes escape sequences, except that unlike echo, you must use \n to
explicitly specify a newline:
$ printf “No filename entered\n”
No filename entered

\n not required in echo

Like its namesake in the C language, printf also uses format specifiers. This is how
you display the shell you are using:
$ printf “My current shell is %s\n” $SHELL
My current shell is /usr/bin/ksh

The %s format used for printing strings acts as a placeholder for the value of $SHELL.
printf here replaces %s with the value of $SHELL. C language users should note the
absence of the parentheses and the comma between the format specifier and its matching arguments.

das76205_Ch02_025-055.indd 39

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

40

2.8 script: Recording Your Session
This command, virtually unknown to many UNIX users, lets you “record” your login
session in a file. script is not included in POSIX, but you’ll find it useful to store in a
file all commands that you invoke, their output and error messages. You can later view
the file. If you are doing some important work and wish to keep a log of all of your
activities, then you should invoke script immediately after you log in:
$ script
Script started, file is typescript
$ _

Another shell!

The prompt returns, and all of your keystrokes (including the one used to backspace)
that you now enter here get recorded in the file typescript. After your recording is
over, you can terminate the session with exit:
$ exit
Script done, file is typescript
$ _

Or use [Ctrl-d]
Back to login shell

You can now view this file with the cat command. script overwrites any previous
typescript that may exist. If you want to append to it, look up the man page to locate
the -a option.
The file created by script contains the control character, [Ctrl-m], at the end
of every line. The cat command won’t show this character, but on your vi editor this
character appears as ^M. Later, you should be able to view it (with cat -v) and remove
it both interactively (with vi) and noninteractively (with sed).

Caution

2.9

Don’t forget to type exit at the end of the recording session. If you don’t, the file typescript
will progressively get bloated and take up much of your disk space. This can easily happen if
you are working on your home computer where you may never log out.

Using Email with mailx
You are probably well-versed in email semantics already. Even if you are totally comfortable using a GUI program like Thunderbird or Outlook Express, it’s necessary to
know one command-line tool that can be used noninteractively in shell scripts: mailx,
the only mail utility that POSIX requires all UNIX systems to support. Using the shell’s
redirection features (like the < and |) and mailx options, we should be able to generate
mail headers and message body on the fly.
An email message is identified by a sender and a recipient, both of which appear
as headers in the message. We’ll save the discussions on these headers for Chapter 11.
Sender and recipient can be on the same or different machines or hosts. Accordingly,
an email address can take the following forms:
henry
henry@saturn
henry@heavens.com

das76205_Ch02_025-055.indd 40

User henry on same host
On a different host
On the Internet

12/13/11 10:43 AM

41

Chapter 2: Becoming Familiar with UNIX Commands

Received mail is deposited in a mailbox. This is simply a text file that may contain
binary attachments in encoded form. When a message has been viewed, it moves from
the mailbox to the mbox. In mail jargon, these files are often referred to as folders.

Note

2.9.1

GUI programs don’t make use of the default mailbox but instead maintain it, along with other
folders that store sent and unsent mail, in a separate directory. Only command-line tools make
use of the mail handling features offered by the UNIX system.

Sending Mail
mailx works in the sending or receiving mode. When you invoke it with the email address of the recipient as argument, the command works in the interactive sending mode.
Key in the subject, and then the message body. Finally press [Ctrl-d] (or a solitary .) to
terminate your input. This is how henry sends mail to charlie:
$ mailx charlie
Subject: New System
The new system will start functioning from next month.
Convert your files by next week - henry
[Ctrl-d] or dot
EOT

charlie is on same host

System indicates end of text

The sent message lands in charlie’s mailbox. If this interactive mode were the only
means of using mailx for sending mail, you’d be better off using Mozilla Thunderbird
or Outlook Express. What makes mailx a “true” UNIX program is that it can be used
noninteractively as well:
mailx -s “New System” charlie < message.txt

The -s option takes care of the subject, and the message body is obtained from the file
message.txt using a shell feature called redirection. No GUI mail program can be
used in this way.
Though POSIX doesn’t require mailx to copy messages to other people, most
versions support the -c (carbon copy) and -b (blind carbon copy) options. Multiple
recipients should be enclosed in quotes:
mailx -s “New System” -c “jpm,sumit” -b andrew charlie < message.txt

This command sends a message to charlie with carbon copies to jpm and sumit, and a
blind carbon copy to andrew.

Note

2.9.2

das76205_Ch02_025-055.indd 41

What makes this method of invocation remarkable is that the subject and recipients need
not be known in advance, but can be obtained from shell variables. The message body could
even come from the output of another program. You can use this feature to design automated
mailing lists.

Receiving Mail
Incoming mail is appended to the mailbox, a text file named after the user-id of the recipient. The mailbox is located in the directory /var/mail (/var/spool/mail in Linux).

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

42

charlie’s mail is appended to /var/mail/charlie. We are often prompted to read the
mailbox by this message from the shell:
You have new mail in /var/mail/charlie

charlie now has to invoke the mailx command in the receiving mode (without using an
argument) to see the mailbox. The system first displays the headers and some credentials
of all incoming mail that’s still held in the mailbox:
$ mailx
mailx version 5.0 Wed Jan 5 16:00:40 PST 2000 Type ? for help.
“/var/mail/charlie”: 5 messages 2 new 5 unread
U 1 andrew@heavens.com
Fri Apr 3 16:38 19/567 “sweet dreams”
U 2 MAILER-DAEMON@heaven Sat Apr 4 16:33 69/2350 “Warning: could not se”
U 3 MAILER-DAEMON@heaven Thu Apr 9 08:31 63/2066 “Returned mail: Cannot”
N 4 henry@heavens.com
Thu Apr 30 10:02 17/515 “Away from work”
>N 5 henry@heavens.com
Thu Apr 30 10:39 69/1872 “New System”
? _
The ? prompt

The pointer (>) is positioned on the fifth message; we call this the current message. charlie can either press [Enter] or the number of the message shown in the second column
to view the message body. The following message is typically seen on charlie’s screen:
Message 5:
>From henry@heavens.com Thu Apr 30 10:39:14 2000
Date: Tue, 13 Jan 2003 10:06:13 +0530
From: “henry blofeld” 
To: charlie@saturn.heavens.com
Subject: New System
The new system will start functioning next month.
Convert your files by next week - henry
? q
Saved 1 message in /home/charlie/mbox
$ _

Quitting mailx with q

As mentioned before, after a message has been seen by the recipient, it moves from the
mailbox to the mbox. This is generally the file named mbox in the user’s home directory
(the directory where the user is placed upon logging in).
2.9.3

mailx Internal Commands
Internal commands are not the sole preserve of the shell. mailx also has its own internal
commands (Table 2.3) that you can enter at the ? prompt. You can see the next message
(if there is one) using the concept of relative addressing. Enter a + to see the next message or a - to display the previous one. A message can also be accessed by its number:
3

das76205_Ch02_025-055.indd 42

Shows message number 3

12/13/11 10:43 AM

43

Chapter 2: Becoming Familiar with UNIX Commands
TA B L E 2 . 3

Internal Commands Used by mailx

Command

Action

+
N

Prints next message
Prints previous message
Prints message numbered N

h

Prints headers of all messages
Deletes message N (the current message if N is not specified)
Undeletes message N (the current message if N is not specified)
Saves current message with headers in flname
(mbox if flname is not specified)
Saves current message without headers in flname
(mbox if flname is not specified)
Forwards mail to user
Replies to sender of message N (the current message if N is not specified)
Quits mailx
Runs UNIX command cmd

dN
uN
s flname
w flname
m user
rN
q
! cmd

Replying to mail charlie can reply to a message by using the r (reply) command,
which switches mailx to the sending mode. Every mail program has a mechanism of
deducing the sender’s details, and consequently, the r command is usually not used
with an address:
? r
To: henry@heavens.com
Subject: Re: File Conversion
I am already through.
[Ctrl-d]
EOT

Sender’s address automatically inserted

Saving Messages Generally all mail commands act on the current message by default.
With the w command, you can save one or more messages in separate files rather than
the default mbox:
w note3
w 1 2 3 note3

Appends current message to note3
Appends first three messages to note3

You can later view these messages with their headers using mailx -f note3.
Deleting Mail To delete a message from the mailbox, use the d (delete) command.
It actually only marks mail for deletion; the mail actually gets deleted only after quitting mailx.

das76205_Ch02_025-055.indd 43

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

44

2.10 passwd: Changing Your Password
The remaining commands in this chapter relate to our UNIX system, and we’ll first
take up the passwd command that changes the user’s password. If your account has a
password that is already known to others, you should change it immediately:
$ passwd
passwd: Changing password for romeo
Enter login password: *******
New password: ********
Re-enter new password: ********
passwd (SYSTEM): passwd successfully changed for romeo

Asks for old password
New password to be
entered twice

passwd changes the password of the user running the command. If everything goes
smoothly, the new password is encrypted (scrambled) by the system and stored in the
file /etc/shadow (/etc/passwd on older systems). This file is used by the system for
authentication every time you log in.
Many systems conduct certain checks on the password string to ensure that you
don’t choose easy-to-remember passwords. Systems often insist on a minimum length
or a mix of letters and numerals. Contrary to popular belief, it is safer to note the password down rather than try to remember it—and then forget it. The author employs the
technique of choosing the name of one of his childhood heroes and then inserting the
digits of the year of birth using a predetermined algorithm. If you adopt this approach,
you need to note down only the algorithm and the name of the hero. It’s not difficult to
obtain the year of birth of our heroes.
Today, many users in a network connect to a UNIX machine using an SSH
program where the password is not used for authentication. Rather, it’s Public Keybased cryptography that authenticates users (and is discussed in Section 11.8). The
/etc/passwd file is used only when users connect through the console, a dumb terminal,
or the telnet command.

Note

A nonprivileged user like you can change only your own password, but the system administrator
(the privileged user) uses the same command to change any user’s password. This is discussed in
Chapter 19. Also, the command may not work on machines using the LDAP network protocol.

2.11 uname: Your Machine’s Name and Operating System
The uname command displays certain features of the operating system running on your
machine. By default, it simply displays the name of the operating system:
$ uname
SunOS

Linux shows Linux

This is the operating system used by Sun Solaris. Using suitable options, you can
display certain key features of the operating system and also the name of the machine.
The output depends on the system you are using.

das76205_Ch02_025-055.indd 44

12/13/11 10:43 AM

45

Chapter 2: Becoming Familiar with UNIX Commands

The Current Release (-r) A UNIX command often varies across versions so much
that you’ll need to use the -r option to find out the version of your operating system:
$ uname -r
5.8

This is SunOS 5.8

This is a machine running SunOS 5.8, the operating system used by the Solaris 8 environment. If a command doesn’t work properly, it could either belong to a different
“implementation” (could be BSD) or a different “release” (may be 4.0, i.e., System V
Release 4 of AT&T).
The Machine Name (-n) Every machine has a name (the hostname), and if your
network is connected to the Internet, this hostname is a component of your machine’s
domain name. The -n option tells you the hostname:
$ uname -n
mercury.heavens.com

The complete domain name

Here, mercury is the hostname and heavens.com is the domain name. Many UNIX networking utilities use the hostname as an argument. To copy files from a remote machine
named mercury, you have to run ftp mercury.

Linux

uname -n may show either the hostname (like mercury) or the complete domain
name (like mercury.heavens.com), depending on the flavor of Linux you are using.
uname and uname -r display the operating system name and version number of the
kernel, respectively:
$ uname
Linux
$ uname -r
2.6.16.60-0.21-smp

Kernel version is 2.6

The first two numbers of the kernel version (here, 2.6) are something every Linux
user must remember. Before installing software, the documentation may require you
to use a kernel that is “at least” 2.4. The same software should run on this machine,
whose kernel version is 2.6.

2.12 who: Know the Users
Let’s take a look at two commands that we introduced in the hands-on session of
Chapter 1. We saw the users logged in with the who command. Let’s now examine the
other columns of the command’s output:
$ who
root
romeo

das76205_Ch02_025-055.indd 45

console
pts/10

Aug 1 07:51
Aug 1 07:56

(:0)
(pc123.heavens.com)

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

46
juliet
project
andrew

pts/6
pts/8
pts/14

Aug 1 02:10
Aug 1 02:16
Aug 1 08:36

(pc125.heavens.com)
(pc125.heavens.com)
(mercury.heavens.com)

The first column shows the user-ids of the five users currently working on the system.
The second column shows the filenames of the devices associated with the respective
terminals. romeo’s terminal has the name pts/10 (a file named 10 in the pts directory). The third, fourth, and fifth columns show the date and time of logging in. The last
column shows the hostname from where the user logged in. Users can log in remotely
to a UNIX system, and all users here except root have logged in remotely from four
different hosts.
One of the users shown in the first column is obviously the user who invoked the
who command. To know that specifically, use the arguments am and i with who:
$ who am i
romeo
pts/10

Aug 1 07:56

(pc123.heavens.com)

UNIX provides a number of tools (called filters) to extract data from command output
for further processing. For instance, you can use the cut command to take out the first
column from the who output and then use this list with mailx to send a message to these
users. The ability to combine commands to perform tasks that are not possible to achieve
using a single command is what makes UNIX so different from other operating systems.
We’ll often combine commands in this text.

2.13 date: Displaying the System Date
The UNIX system maintains an internal clock meant to run perpetually. When the system
is shut down, a battery backup keeps the clock ticking. This clock actually stores the
number of seconds elapsed since the Epoch: January 1, 1970. A 32-bit counter stores
these seconds (except on 64-bit machines), and the counter will overflow sometime
in 2038.
You can display the current date with the date command, which shows the date
and time to the nearest second:
$ date
Mon Aug 11 17:04:30 GMT 2003

The command can also be used with suitable +-prefixed format specifiers as arguments.
For instance, you can print only the month, using the format +%m:
$ date +%m
08

das76205_Ch02_025-055.indd 46

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

47

or the month name:
$ date +%h
Aug

or you can combine them in one command:
$ date +”%h %m”
Aug 08

When you use multiple format specifiers, you must enclose them within quotes (single
or double) and use a single + symbol as a prefix. Here’s a useful list of the other
format specifiers:
d—The day of the month (1 to 31).
y—The last two digits of the year.
H, M and S—The hour, minute, and second, respectively.
D—The date in the format mm/dd/yy.
T—The time in the format hh:mm:ss.

Note

You can’t change the date as an ordinary user, but the system administrator uses the same
command with a different syntax to set the system date! This is discussed in Chapter 19.

2.14 stty: When Things Go Wrong
Different terminals have different characteristics, and your terminal may not behave in
the way you expect it to. Sometimes you may want to change the settings before you
run a program. The stty command changes terminal settings but also displays them
when used with the -a option:
$ stty -a
speed 38400 baud; rows = 24; columns = 80; ypixels = 0; xpixels = 0;
intr = ^c; quit = ^\; erase = ^?; kill = ^u;
eof = ^d; eol = ; eol2 = ; swtch = ;
start = ^q; stop = ^s; susp = ^z; dsusp = ^y;
isig icanon -xcase echo echoe echok -echonl -noflsh
-tostop echoctl -echoprt echoke -defecho -flusho -pendin iexten

stty shows the settings of several keywords in this trimmed output. The first line shows
38,400 as the baud rate (the speed) of the terminal. The other keywords take two forms:
• keyword = value
• keyword or -keyword. The - prefix implies that the option is turned off.
Let’s now understand the significance of some of these keywords and then use
stty to change the settings.

das76205_Ch02_025-055.indd 47

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

48
2.14.1

Changing the Settings
Interrupting a Command (intr) Sometimes you might want to interrupt a program
before it completes. The stty output shows intr as the interrupt key, which here is
set to ^c (a caret and c signifying [Ctrl-c]). Pressing this key on this machine should
terminate a program. You can define a different key for this function:
stty intr DEL

The [Delete] key

To revert to the original setting, use stty intr \^c. The \ is often used in UNIX to
emphasize that the character following it needs to be interpreted differently. Here, it
suggests that ^ must not be interpreted as the caret character, but as a component of
the [Ctrl-c] sequence. We’ll see more of the \ when we take up the shell in Chapter 6.
Changing the End-of-File Key (eof) When using mailx, you used [Ctrl-d] to terminate input. Many commands like cat and wc also use the eof or end-of-file character, which
stty understands as the keyword eof. The stty output shows eof set to ^d ([Ctrl-d]).
You can change this setting also in the manner described previously.
Backspacing (erase) Backspacing is controlled by the erase character, which is set
to ^?, the key labeled [Backspace]. Sometimes backspacing may not work at all and
instead produce a series of ^H characters every time you press the key:
$ password^H^H^H

[Backspace] pressed three times

This often happens when you log on to a remote machine whose terminal settings are
different from your local one. Try using [Ctrl-h] or [Delete] or explicitly assign the
[Backspace] key in this way:
stty erase [Backspace]

Press the [Backspace] key after erase

Suspending a Job (susp) Modern shells allow you to suspend a job and then resume
it later. stty shows [Ctrl-z] as the stop character. When a command is running you can
press this key, but do so only after you have read Section 7.11.
Entering a Password through a Shell Script (echo) Shell programmers often
manipulate echo to let shell programs accept passwords without displaying them on
screen. Here it is on, but we can turn it off with stty -echo. You should turn it off after
the entry is complete by using stty echo.
When Everything Else Fails (sane) stty is also used to set the terminal characteristics to values that will work on most terminals. Use stty sane or the reset command
to restore sanity to the terminal.
These key functions are summarized in Table 2.4. Note that the default key sequences (like eof and interrupt) could be different on your system. In any case, you are
advised against tampering with too many settings.

das76205_Ch02_025-055.indd 48

12/13/11 10:43 AM

49

Chapter 2: Becoming Familiar with UNIX Commands
TA B L E 2 . 4

Tip

2.15

stty Settings and Keyboard Commands to Try When Things Go Wrong

stty Keyword

Typical Setting

Function

erase
interrupt
eof

[Ctrl-h]
[Ctrl-c] or [Delete]
[Ctrl-d]

stop
start
kill
quit

[Ctrl-s]
[Ctrl-q]
[Ctrl-u]
[Ctrl-\]

susp

[Ctrl-z]

echo

-

sane

-

Erases text
Interrupts a command
Terminates input to a program that expects
input from the keyboard
Stops scrolling of display and locks keyboard
Resumes scrolling of display and unlocks keyboard
Kills command line without executing it
Kills running command but creates a core file
containing the memory image of the program
Suspends process and returns shell prompt;
use fg to resume job
Enables display to echo keyboard input
(-echo to disable)
Restores terminal to normal status

Keep these two keys in mind: [Ctrl-c], the interrupt key, used to interrupt a running program,
and [Ctrl-d], the eof key, used to signify the end of terminal input to a program that’s expecting input from the terminal.

The X Window System
Finally, let’s briefly examine an alternative to the command line interface. Every UNIX
system supports a graphical user interface (GUI), which generically is known as the X
Window system. Like the ubiquitous Microsoft Windows, X (i.e., X Window) displays
every application in a separate window. It uses the mouse to invoke programs, display
menus, select options, and handle cut-copy-paste operations. Because X uses a bit-mapped
display (where every pixel on the screen is handled individually), Web browsers like
Mozilla and Konqueror must run under X.
X was originally developed to work in a network, and Section 11.12 examines the
networking features of X. But X also supports a host of applications that can be used
in standalone mode. However, the use of X is strongly discouraged for handling tasks
that are better handled by the command line. Alternatively, you can use this command
line from a window running under X. This application is called the terminal emulator,
which along with a file management program, is discussed next.
Unlike Microsoft Windows, the look-and-feel of the X Window system varies
widely across UNIX and Linux systems. For some time, it appeared that vendors had
finally standardized on the Common Desktop Environment (CDE). Eventually, Solaris
abandoned CDE, and Linux systems have several alternatives (like GNOME or KDE)
for you to choose from.

das76205_Ch02_025-055.indd 49

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

50
2.15.1

The Terminal Emulator
X supports a program called the terminal emulator, which runs a shell in a window
(Fig. 2.4). You can enter every UNIX command from the shell prompt of this window.
UNIX commands use a character-based display, but you can also invoke any graphical
X program from the emulator window. This is how you run Mozilla Firefox:
firefox &

Run all X programs in the background with &

The original emulator program is xterm. CDE offered dtterm, but today we have better
emulator programs to choose from. The one shown in Fig. 2.4 is the konsole program
offered on SuSE Linux. Every emulator features a menu and a scrollbar that allows you
to recall previous commands and their output. The exit command that kills your login
session on a character terminal kills the window and its associated shell here.
Almost every X application allows cut-copy-paste operations. To copy text
from one window to another, first highlight the text with the left mouse button; the
text automatically gets copied to a buffer. Now select the other window, click on the
desired location, and then click on the middle button (or both buttons simultaneously
when using a two-button mouse). The copied text is pasted on this window. Using this
technique, you can re-execute a long command line on a different window or copy text
from a Web page to your vi editor. Usually cut-copy-paste operations are also available as menu options.

FIGURE 2.4

das76205_Ch02_025-055.indd 50

A Terminal Emulator Program

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands

51

You can have several terminal emulators (apart from several programs) on your
desktop, and you can invoke a separate application in each one of them. You can also
switch from one application to another without quitting any of them.

Tip

2.15.2

If you have difficulty in copy-paste operations using the technique described here, you can
use the window menu, which also offers options to do the same work. Often, the keys are the
same ones used in Microsoft Windows—[Ctrl-c] for copying and [Ctrl-v] for pasting.

The File Manager
We use files all the time, copying, moving and deleting them several times a day. Every
X implementation offers a file management program that can perform these tasks. A file
manager can also be used to view file contents and execute programs. Windows offers a
similar application—Windows Explorer. The file manager on the CDE is dtfile, which
is shown in Fig. 2.5. Linux users may use Konqueror instead. However, your system
may contain other file managers.
Using menu options, you can create and remove directories. Try creating some.
To copy or move files from one directory to another, you need to work with two windows of the same program. Look up the menu option that splits a window. Every file
is represented by an icon, and you select it by clicking it with the mouse. You can also
select multiple files by pressing [Ctrl] and then clicking on each icon. To select all files,

FIGURE 2.5

das76205_Ch02_025-055.indd 51

Two Views of the dtfile File Manager

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

52

use the option offered by the menu; it often is [Ctrl-a]. You can now drag the files by
keeping the left mouse button pressed and drop them to their new location by releasing
the button. Files can thus be copied and moved in this way.
In Chapter 3, you’ll use the mkdir, rmdir, cp and mv commands for file and
directory handling. You’ll see how effortlessly you can work with groups of files and
directories using these commands. However, it’s good to know the X techniques now
because that will help you appreciate the power of the UNIX command line interface
later. The limitations of the X method of doing things will soon become apparent.

SUMMARY
UNIX commands are case-sensitive but are generally in lowercase. They need not have
any specific extensions. Commands for general use are located in the directories /bin
and /usr/bin. The shell variable, PATH, specifies the search list of directories for locating commands.
The shell treats a command either as external when it exists on disk, or internal
when it is built into the shell. Commands like man and mailx also have their own internal commands.
The command line comprises the command and its options and arguments.
Commands and arguments are separated by whitespace. Multiple commands can be
delimited with a ;, and a command sequence can be split into multiple lines.
Use the man command to look up the documentation for a command, a configuration file, or a system call. Most commands are found in Section 1. You’ll find system
calls and library functions in Sections 2 and 3.
echo displays a message on the screen. It supports escape sequences (like \c and
\007). The command has portability problems, the reason why printf should be used.
printf also supports format specifiers (like %d).
script logs all user activities in a separate file named typescript.
A mail message is saved in a text file called mailbox. Mail is moved to the mbox
after it is viewed. mailx, a command-line mail program, can be used interactively and
also noninteractively from shell scripts.
date displays any component of the system date and time in a number of formats.
passwd changes a user’s password. The system administrator can change the system
date and the password of any user.
uname reveals details of your machine’s operating system (-r and -s). It also
displays the hostname (-n) that is used by networking commands.
who displays the users working on the system.
stty displays and sets various terminal attributes. It defines the key that interrupts
a program (intr), suspends a job (susp), and marks the end-of-file (eof). stty sane
sets the terminal to some standard values.
The X Window system provides a graphical user interface (GUI) for users to run
programs that involve graphics. X also provides several applications including a terminal
emulator and a file management program.

das76205_Ch02_025-055.indd 52

12/13/11 10:43 AM

53

Chapter 2: Becoming Familiar with UNIX Commands

SELF-TEST
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24

Enter a : and press [Enter]. Next run type :. What do you conclude?
UNIX commands must be in lowercase and must not have extensions. True or
false?
Name three UNIX commands whose names are more than five characters long.
Find out whether these commands are internal or external:
echo, date, pwd, and ls.
If two commands with the same filename exist in two directories in PATH, how
can they be executed?
How is the current directory indicated in PATH?
How many options are there in this command? ls -lut chap01 note3
If you find yourself using options preceded by two hyphens (like --all), which
flavor of UNIX could you be using?
What is the name given to the command and its options and arguments?
How do you find out the version number of your operating system?
Why are the directories /bin and /usr/bin usually found first in PATH?
What is whitespace? Explain the treatment the shell metes out to a command
that contains a lot of whitespace.
Do you need to wait for a command to finish before entering the next one?
Why doesn’t this command run in the way it is meant to?
printf “Filename: %s\n”, fname
What is a pager? Name the two standard pagers used by man.
You located the string crontab in a man page by searching with /crontab[Enter].
How do you find out the other occurrences of this string in the page?
You don’t know the name of the command that could do a job. What do you do?
How do you find out the users who are idling from the man documentation
of who?
What is the difference between the mailbox and mbox?
The passwd command didn’t prompt for the old password. When do you think
that can happen? Where is the password stored?
Can you change the system date with the date command?
Enter the uname command without any arguments. What do you think the output
represents?
How will you record your login session in the file foo?
Interpret the following output of who am i:
romeo

2.25
2.26

das76205_Ch02_025-055.indd 53

pts/10

Aug 1 07:56

(pc123.heavens.com)

How do you determine the erase, kill and eof characters on your system?
You suddenly find your keyboard is displaying uppercase letters even though
your [CapsLock] key is set properly. What should you try?

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

54

EXERCISES
2.1
2.2
2.3

2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11

2.12

Enter a # before a command and press [Enter]. What do you see, and how do
you think you can take advantage of the behavior?
Name three major differences between UNIX commands and Windows programs.
A program file named foo exists in the current directory, but when we try to
execute it by entering foo, we see the message foo: command not found.
Explain how that can happen.
If a command resides in a directory which is not in PATH, there are at least two
ways you can still execute it. Explain.
Where are the commands used by the system administrator located?
You won’t find the cd command either in /bin or /usr/bin. How is it executed
then?
If you find the echo command in /bin, would you still call it an external
command?
Is an option also an argument? How many arguments are there in this command?
cat < foo > bar
Why shouldn’t you have a filename beginning with a -?
Reduce the number of keystrokes to execute this command:
tar -t -v -f /dev/fd0.
Look up the tar man page to find out whether the command
tar -cvfb 20 foo.tar *.c is legitimate or not. Will the command work
without the - symbol?
Both of the following commands try to open the file foo, but the error messages
are a little different. What could be the reason?
$ cat foo
cat: foo: No such file or directory
$ cat < foo
bash: foo: No such file or directory

2.13
2.14
2.15

Invoke the commands echo hello dolly and echo “hello dolly” (three
spaces between hello and dolly). Explain the difference in command behavior.
What does the secondary prompt look like, and when does it appear?
What do the | and the three dots in the SYNOPSIS section of these man pages
indicate as shown below?
/usr/xpg4/bin/tail [ -f | -r ]
/usr/bin/ls [ -aAbcCdfFgilLmnopqrRstux1 ] [ file ... ]

2.16

2.17
2.18

das76205_Ch02_025-055.indd 54

If a command, filename, and a system call have the same name and are available
in Sections 1, 5, and 2 respectively, how will you display the man pages of each
one of them?
Your system doesn’t have the apropos command. What will you do?
The command echo “Filename: \c” didn’t place the cursor at the end of
the line. How will you modify the command to behave correctly if your shell
is (i) Bash, (ii) any other shell?

12/13/11 10:43 AM

Chapter 2: Becoming Familiar with UNIX Commands
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27

2.28
2.29

das76205_Ch02_025-055.indd 55

55

What is an escape sequence? Name three escape sequences used by the echo
command, and explain the significance of each.
Use printf to find out the hex and octal values of 255.
Run ps, then the script command, and then run ps again. What do you notice?
In what way is the mailx command superior to a GUI program like Netscape or
Mozilla?
Can you have the same user-id more than once in the who output?
Both your local and remote machines use identical versions of UNIX. How do
you confirm whether you are logged in to a remote machine or not?
Which command does the nonprivileged user use to change the system date
and time?
Display the current date in the form dd/mm/yyyy.
You need to accept a secret code through a shell script. What command will you
run in the script to make sure that your keyboard input is not displayed? How
do you then revert to the normal setting?
Explain why it is possible to key in the next command before the previous command has completed execution.
What will you do to ensure that [Ctrl-c] interrupts a program? Will it work the
next time you log in?

12/13/11 10:43 AM

CHAPTER

3

The File System

U

NIX looks at everything as a file, and any UNIX system has thousands of
files. For convenience, we make a distinction between ordinary files and directories that house groups of files. In this chapter, we’ll create directories, navigate the
file system, and list files in a directory. We’ll also examine the structure of the standard
UNIX file system.
In addition, we’ll create, copy, move, and delete files, and we will learn how these
actions affect the directory. Some commands exhibit recursive behavior by descending a
directory structure to perform some action. Because we frequently encounter Windows
systems, we need to be able to move files between UNIX and Windows systems. As
Internet users, we also need to handle compressed files that we download. However, we
don’t tamper with the major file attributes in this chapter.

Objectives
• Understand the initial categorization of files into ordinary, directory, and device.
• Learn the hierarchical structure of the file system, and how UNIX organizes its
own data.
• Understand the significance of the home directory and current directory.
• Create and remove directories with mkdir and rmdir.
• Navigate the file system with cd and pwd.
• Become aware of the significance of absolute and relative pathnames.
• List files with ls.
• Copy, rename, and delete files with cp, mv, and rm.
• View text files with cat and more.
• Create and edit a file with pico.
• Count the number of lines, words, and characters with wc.
• Learn how UNIX handles printing, and use lp and lpr.
• Display the nonprintable characters in a file with od.
• Convert between UNIX and DOS formats with unix2dos, dos2unix and the Tofrodos
package.
• Compress files with gzip and create archives comprising multiple files with tar.
• Perform both compressing and archiving with zip.

56

das76205_Ch03_056-091.indd 56

12/13/11 10:43 AM

Chapter 3: The File System

57

3.1 The File
The file is a container for storing information. As a first approximation, we can treat
it simply as a sequence of characters. UNIX doesn’t impose any structure for the data
held in a file. It’s for you to do that to suit your programs. A file doesn’t contain an
end-of-file (eof) mark, but that doesn’t prevent the kernel from knowing when to stop
reading the file. Neither the file’s name nor its size is stored in the file. File attributes
are kept in a separate area of the hard disk, not directly accessible to users, but only
to the kernel.
The shell is a file and so is the kernel. UNIX treats directories and devices like the
hard disk, DVD-ROM, and printer as files as well. As an initial exercise, let’s understand
a file as being of three types:
• Ordinary file—Also known as a regular file. It contains only data as a stream of
characters.
• Directory file—A folder containing the names of other files and subdirectories as
well as a number associated with each name.
• Device file—This represents a device or peripheral. To read or write a device, you
have to perform these operations on its associated file.
There are other types of files, but we’ll stick to these three for the time being. We need
to make this distinction between file types because the significance of a file’s attributes
often depends on its type. Execute permission for an ordinary file means something quite
different from that for a directory. You can’t directly put something into a directory file,
and a device file isn’t really a stream of characters. Some commands work with all file
types, but some don’t.
3.1.1

Ordinary (Regular) File
An ordinary file is the most common file type containing a stream of data. This type
of file can be further divided into two types:
• Text file
• Binary file
A text file contains only printable characters. All C and Java program sources, and shell
and Perl scripts, are text files. A text file contains lines where each line is terminated
with the linefeed (LF) character, also known as newline. When you press [Enter] while
inserting text in a text editor like vi or pico, the LF character is appended to every
line. You won’t see this character normally, but the cat -e command makes it visible
(by showing a $ at the end of each line). Also, the od command makes all characters
in a file visible.
A binary file, on the other hand, contains both printable and nonprintable characters that cover the entire ASCII range (0 to 255). Most UNIX commands are binary
files, and the object code and executables that you produce by compiling C programs
are also binary files. Picture, sound, and video files are binary files as well (with few
exceptions). Displaying such files with a simple cat command produces unreadable
output and may even disturb your terminal’s settings.

das76205_Ch03_056-091.indd 57

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

58
3.1.2

Directory File
A directory contains no data as such, but maintains some details of the files and subdirectories that it contains. The UNIX file system is organized with a number of directories and subdirectories. You can also create them when you need to group a set of files
pertaining to a specific application.
A directory file contains an entry for every file and subdirectory that it houses.
If you have 20 files in a directory, there will be 20 entries in the directory. Each entry
has two components:
• The filename.
• A unique identification number for the file or directory (called the inode number).
If a directory bar contains an entry for a file foo, we commonly (and loosely) say that
the directory bar contains the file foo. Though we’ll often be using the phrase “contains
the file” rather than “contains the filename,” you must not interpret the statement literally. A directory contains the filename and not the file’s contents.
You can’t, however, write a directory file, but you can perform some action that
makes the kernel write a directory. For instance, when you create or remove a file, the
kernel automatically updates its corresponding directory by adding or removing the entry
(inode number and filename) associated with the file.
The name of a file can only be found in its directory. The file itself doesn’t contain its own name
or any of its attributes, like its size or time of last modification.

Note

3.1.3

3.2

Device File
You’ll also be printing files, installing software from DVD-ROMs, or backing up files
to tape. All of these activities are performed by reading or writing the file representing
the device. For instance, when you restore files from tape, you read the file associated
with the tape drive.
A device file is indeed special; it’s not really a stream of characters. In fact, it
doesn’t contain anything at all. You’ll soon learn that every file has some attributes that
are not stored in the file but are stored elsewhere on disk. The attributes of a device
file entirely govern the operation of the device. The kernel identifies a device from its
attributes and then uses them to operate the device.
Now that you understand the three types of files, you shouldn’t feel baffled by
subsequent use of the word in the book. The term file will often be used in this book to
refer to any of these types, though it will mostly be used to mean an ordinary file. The
real meaning of the term should be evident from its context.

What’s in a (File)name?
On most UNIX systems today, a filename can consist of up to 255 characters. Files may
or may not have extensions, and can consist of practically any ASCII character except
the / and the NULL character (ASCII value 0). As a general rule you should avoid using unprintable characters in filenames. Further, since the shell has a special treatment

das76205_Ch03_056-091.indd 58

12/13/11 10:43 AM

Chapter 3: The File System

59

for characters like $, `, ?, *, & among others, it is recommended that only the following
characters be used in filenames:
• Alphabetic characters and numerals.
• The period (.), hyphen (-), and underscore (_).
UNIX imposes no restrictions on the extension, if any, that a file should have.
A shell script doesn’t need to have the .sh extension even though it helps in identification.
But the C compiler expects .c program files and Java expects .java. DOS/Windows
users must also keep these two points in mind:
• A filename can comprise multiple embedded dots; a.b.c.d.e is a perfectly valid
filename. Moreover, a filename can also begin with a dot or end with one.
• UNIX is sensitive to case; chap01, Chap01 and CHAP01 are three different filenames,
and it’s possible for them to coexist in the same directory. This is, however, not true
for Mac OS X, the operating system used by Apple Macintosh.

Caution

3.3

Never use a - at the beginning of a filename. You could have a tough time getting rid of it!
A command that uses a filename as an argument often treats it as an option and reports errors.
For instance, if you have a file named -z, cat -z won’t display the file but will interpret it as
an invalid option. There’s a workaround, though: embed the filename in a pathname. Both
cat ./-z and rm test/-z would work on most systems.

The File System Hierarchy
All files in UNIX are organized in a hierarchical (an inverted tree) structure (Fig. 3.1).
This hierarchy has a top called root, which serves as the reference point for all files.
root is actually a directory that is represented by a / (frontslash). Don’t mix up the root
directory with the user-id root, which is used by the system administrator to log in. In this
text, we’ll be using both the name “root” and the symbol / to represent the root directory.
The root directory (/) has a number of subdirectories under it. These subdirectories
have more subdirectories and other files under them. For instance, home is a directory
under root, and romeo is yet another directory under home. login.sql is presumably
an ordinary file under romeo. Every hierarchy contains parent-child relationships, and
we can conveniently say that romeo is the parent of login.sql, home is the parent of
romeo, and / (root) is the parent of home.
We can specify the relationship login.sql has with root by a pathname:
/home/romeo/login.sql. The first / represents the root directory and the remaining /s
act as delimiters of the pathname components. This pathname is appropriately referred
to as an absolute pathname because by using root as the ultimate reference point we
can specify a file’s location in an absolute manner. To view this file, we need to use
cat /home/romeo/login.sql.
When you specify a file using absolute pathnames, you have a mechanism for
identifying a file uniquely. No two files in a UNIX system can have identical absolute
pathnames. You can have two files with the same name, but in different directories; their
pathnames will also be different. Thus, the file /home/romeo/progs/fork.c can coexist
with the file /home/romeo/safe/fork.c.

das76205_Ch03_056-091.indd 59

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

60
FIGURE 3.1

The UNIX File System Tree
/

bin

dsk

dev

fd0

home

tmp

sbin

etc

usr

var

lp0
romeo

progs

lib

juliet

login.sql

bin

sbin

lib

local

.profile

It’s obvious that the parent is always a directory. home and romeo are both directories as they are both parents of at least one file or directory. An ordinary or device file
like login.sql can’t be the parent of another file.

Caution

3.4

We don’t always use absolute pathnames in our command lines. When you access a file in
the current directory or in its subdirectories, the first / should be dropped. The command
cat /progs/foo.c is different from cat progs/foo.c.

The UNIX File System
Now let’s take a cursory look at the structure of the UNIX file system. This structure
had changed constantly over the years until AT&T proposed its SVR4 release. Though
vendor implementations vary in detail, broadly the SVR4 structure has been adopted by
most vendors. Fig. 3.1 shows a heavily trimmed structure.
For our initial comprehension, we’ll stick to the directories that follow. It helps,
from the administrative point of view at least, to view the entire file system as comprising two groups. The first group contains the files that are made available during system
installation:
• /bin and /usr/bin These are the directories where all the commonly used UNIX
commands (binaries, hence the name bin) are found. Note that the PATH variable
always shows these directories in its list.

das76205_Ch03_056-091.indd 60

12/13/11 10:43 AM

61

Chapter 3: The File System

• /sbin and /usr/sbin If there’s a command that you can’t execute but the system
administrator can, then it would probably be in one of these directories. You won’t
be able to execute most (some, you can) commands in these directories. Only the
system administrator’s PATH shows these directories.
• /etc This directory contains the configuration files of the system. You can change
a very important aspect of system functioning by editing a text file in this directory.
Your login name and password are stored in files /etc/passwd and /etc/shadow.
• /dev This directory contains all device files. These files don’t occupy space on
disk. There could be more subdirectories like pts, dsk, and rdsk in this directory.
• /lib and /usr/lib These directories contain all library files in binary form. You
need to link your C programs with files in these directories.
• /usr/include This directory contains the standard header files used by C programs. The statement #include  used in most C programs refers to
the file stdio.h in this directory.
• /usr/share/man This is where the man pages are stored. There are separate
subdirectories here (like man1, man2, etc.) that contain the pages for each section.
For instance, the man page of ls can be found in /usr/share/man/man1, where
the 1 in man1 represents Section 1 of the UNIX manual. These subdirectories may
have different names on your system (like sman1, sman2, etc., in Solaris).
Users also work with their own files; they write programs, send and receive mail, and
create temporary files. These files are available in the second group:
• /tmp The directories where users are allowed to create temporary files. These
files are wiped away regularly by the system.
• /var The variable part of the file system. Contains all of your print jobs and your
outgoing and incoming mail.
• /home On many systems, users are housed here. romeo would have his home
directory in /home/romeo. However, your system may use a different location for
home directories.
On a busy system, it’s in directories belonging to the second group that you could
experience rapid depletion of available disk space. You’ll learn later to house some
of these directory structures on separate file systems so that depletion of space (and
corruption) in one file system doesn’t affect other file systems.

3.5

Using Absolute Pathnames with Commands
Absolute pathnames have universal application. They can be used either with the command name or its argument if it represents a filename. To illustrate the latter, we often
use the command
cat /etc/passwd

to look up the passwd file in the directory /etc. The command will also work if we use
the absolute pathname for cat as well:
/bin/cat /etc/passwd

das76205_Ch03_056-091.indd 61

Assuming that cat exists in /bin

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

62

We don’t need to use the absolute pathname with cat because it is found in /bin or
/usr/bin, both of which are standard components of PATH. But there are two possible
situations when a command must be used with an absolute pathname:
• If firefox is available in /usr/local/bin, and this directory is not included in
PATH, then we need to use /usr/local/bin/firefox.
• A command sometimes occurs in two directories, both of which could be in PATH.
For instance, on Solaris systems two versions of grep are found in /usr/bin and
/usr/xpg4/bin, and if /usr/bin occurs prior to /usr/xpg4/bin on our system:
PATH=/bin:/usr/bin:/usr/xpg4/bin:.

then we need to use /usr/xpg4/bin/grep to take advantage of the features of
grep that conform to the X/Open Portability Guide.
However, if you are frequently accessing programs in a certain directory, it often makes
sense to include the directory itself in PATH. The technique of doing that is shown in
Section 8.3.

3.6 The HOME Variable and ~: The Home Directory
When you log onto the system, say using romeo as the user-id, you are placed in your
home directory. The shell variable HOME maintains the absolute pathname of this
directory:
$ echo $HOME
/home/romeo

The system administrator sets the home directory for a user in /etc/passwd at the time
of opening a user account. On many UNIX systems, home directories are maintained
in /home, but your home directory could be located differently (say, in /export/home).
It’s often convenient to refer to a file foo located in the home directory as $HOME/foo.
Most shells (except Bourne) also use the ~ symbol to refer to the home directory. It
is a little tricky to use because it can refer to any user’s home directory and not just your
own. For instance, you can also access $HOME/foo as ~/foo. If user juliet has the same
file in her home directory, then romeo can access it as ~juliet/foo. The principle is this:
A tilde followed by / (like ~/foo) refers to one’s own home directory, but when followed
by a string (~juliet) refers to the home directory of that user represented by the string.
In your shell scripts, never refer to files in your home directory or in its subdirectories by their
absolute pathnames. Use $HOME/progs or ~/progs rather than /home/romeo/progs. This lets
you move the scripts to a different system where the home directory is different, say, /u2/romeo,
because $HOME and ~ evaluate differently.

Tip

3.7 pwd and cd: Navigating the File System
Just as a file has a location, UNIX makes users believe that they too are placed in a
specific directory. It also allows you to move around in the file system. At any instant

das76205_Ch03_056-091.indd 62

12/13/11 10:43 AM

Chapter 3: The File System

63

of time, you are located in a directory known as the current directory. The pwd (print
working directory) command displays the absolute pathname of this directory:
$ pwd
/home/romeo

Navigation is performed with the cd (change directory) command. This command can
be used either with or without an argument. When used with one, it switches to the
directory:
$ pwd
/home/romeo
$ cd progs
$ pwd
/home/romeo/progs

Switches to the progs directory

The command cd progs means: “Change your subdirectory to progs under the current
directory.” We didn’t use an absolute pathname here (cd /home/romeo/progs) because
that would require more keystrokes.
cd can also be used without an argument; it simply returns you to your home
directory:
$ pwd
/home/romeo/progs
$ cd
$ pwd
/home/romeo

cd used without arguments
reverts to the home directory

Attention, DOS users! cd invoked without an argument doesn’t display the current directory. We often use this form when we have moved away (to say, /home/juliet or
~juliet) and now want to make an immediate return:
$ cd /home/juliet
$ pwd
/home/juliet
$ cd
$ pwd
/home/romeo

Or cd ~juliet

Returns to home directory

The cd command can sometimes fail if you don’t have proper permissions to access the
directory. This doesn’t normally happen unless you deliberately tamper with the directory’s permissions. Navigation with the cd command using mostly absolute pathnames
is illustrated in Fig. 3.2. Directory permissions are examined in Section 4.4.

Note

das76205_Ch03_056-091.indd 63

Unlike in DOS, when cd is invoked without arguments, it simply reverts to its home directory.
It doesn’t show you the current directory!

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

64
FIGURE 3.2

Navigation with the cd Command

/

cd /

bin
(/bin)

cd /home

home
(/home)

cd /home/juliet

cd /bin
romeo
(/home/romeo)

juliet
(/home/juliet)

cd docs

progs
(/home/romeo/progs)

doc
(/home/romeo/docs)

cd /home/juliet/docs

docs
(/home/juliet/docs)

Note: Absolute pathname indicated in parentheses

3.8

Relative Pathnames (. and ..)
The command cd progs worked because progs resides in the current directory. This
command will also work if progs contains a directory scripts under it:
cd progs/scripts

progs is in current directory

A file is looked for in the current directory if its pathname doesn’t begin with a /. The
pathname progs/scripts is not an absolute one because it doesn’t begin with a /. UNIX
allows the use of two symbols in pathnames that use the current and parent directory
as the reference point:
• . (a single dot) This represents the current directory.
• .. (two dots) This represents the parent directory.
Pathnames that begin with either of these symbols are known as relative pathnames.
The command cd progs is really a synonym for cd ./progs, so progs/scripts is
also a relative pathname. The usefulness of the . becomes obvious when we execute

das76205_Ch03_056-091.indd 64

12/13/11 10:43 AM

Chapter 3: The File System

65

our own cat program that exists in the current directory. Since cat also exists in /bin,
we need to use ./cat foo to run our version of cat rather than the standard one.

Tip

Make sure that the name of a shell script or C program written by you doesn’t conflict with
one in the UNIX system by using either type, which, or whereis with the program name you
have developed. If you find that a program of the same name exists in another directory in
PATH, then either change the name of your program or run it as ./foo.

Now let’s turn our attention to the .. for framing relative pathnames. In a previous
example we used cd /home/juliet when our current directory was /home/romeo. We
could easily have used a relative pathname here:
$ pwd
/home/romeo
$ cd ../juliet
$ pwd
/home/juliet
$ cd ..
$ pwd
/home

Moves one level up and then down

Moves one level up

Note the second invocation of cd uses .. as a single argument. We often use this compact method to ascend the hierarchy. You can also combine any number of such sets of
.. separated by /s:
$ pwd
/home/romeo/pis
$ cd ../..
$ pwd
/home

Moves two levels up

The significance of the pathname components changes here; the .. on the right of the
/ is the parent of the .. on the left. Contrast this with cd bar1/bar2 where bar1 is the
parent of bar2. The use of relative pathnames using .. is depicted in Fig. 3.3.
The . and .. can also be gainfully used with commands that use a directory name
as argument. Consider these sequences that use the cp command for copying a file:
cp /home/juliet/addressbook.sam .
cp addressbook.sam ..

In the first case, the file is copied to the current directory. The second command copies
the same file from the current directory to the parent directory.

Note

das76205_Ch03_056-091.indd 65

Absolute pathnames can get very long if you are located a number of “generations” away
from root. Whether you should use one depends solely on the relative number of keystrokes
required. Even though the relative pathname required fewer key depressions in all of these
examples, that may not always be true.

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

66
FIGURE 3.3

Navigation with Relative Pathnames

/

home
(/home)
cd ..

cd ../..
juliet
(/home/juliet)

romeo
(/home/romeo)

cd ../../juliet/progs

progs
(/home/romeo/progs)

progs
(/home/juliet/progs)

docs
(/home/juliet/docs)

cd ../docs

3.9 mkdir: Making Directories
The mkdir (make directory) command creates one or more directories. Let’s use this
command to create one or more directories in our home directory:
mkdir patch
mkdir patch dbs doc

Three directories created

The second example provides the first justification for using commands rather than GUI
programs. Can you use a Windows Explorer-type program to create three directories
as effortlessly as you do with mkdir? That’s not all: a single invocation of mkdir can
even create a directory tree. Here’s how you create both a directory progs and two
subdirectories, include and lib:
mkdir progs progs/include progs/lib

Creates the directory tree

Note that the sequence of arguments is important; first progs has to be created and
then its subdirectories. (There’a a better method though, but that is left as a Self-Test
exercise for you.)

das76205_Ch03_056-091.indd 66

12/13/11 10:43 AM

Chapter 3: The File System

67

Sometimes the system refuses to create a directory:
$ mkdir test
mkdir: Failed to make directory “test”; Permission denied

This can happen due to these reasons:
• The directory test may already exist.
• There may be an ordinary file by that name in the current directory.
• The permissions set for the current directory don’t permit the creation of files and
directories by the user. You’ll most certainly get this message if you try to create
a directory in /bin, /etc, or any other directory that houses the UNIX system’s
files. Also, user romeo can’t create files in the directory structure of user juliet
unless the latter has explicitly granted that permission.
• There may be no space left on the file system to permit creation of files and
directories.
We’ll take up file and directory permissions in Chapter 4.

3.10 rmdir: Removing Directories
The rmdir (remove directory) command removes empty directories. We can reverse
the previous actions of mkdir like this:
rmdir patch
rmdir patch dbs doc

Directories must be empty

We can also delete the directory tree that we created with mkdir. This time we must
delete the subdirectories before the parent directory:
rmdir progs/lib progs/include progs

Removes the directory tree

A directory can also be removed with the rm command. If you are using rmdir, the
following conditions need to be fulfilled:
• The directory is empty. However, the rm command can remove a nonempty directory.
• The user’s current directory is above the directory.
Let’s test these rules by trying to delete the directory tree progs that was created with
mkdir:
$ rmdir progs
rmdir: `progs’: Directory not empty
$ cd progs ; pwd
/home/romeo/progs
$ rmdir include lib
$ rmdir .
rmdir: `.’: Invalid argument

das76205_Ch03_056-091.indd 67

Contains include and lib

progs is now empty
but you can’t delete it ...

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

68
$ cd .. ; pwd
/home/romeo
$ rmdir progs
$ _

unless you move up and ...
run rmdir from here

The mkdir and rmdir commands work only in directories owned by the user. A user is
the owner of her home directory, so she can use these commands in her home directory
tree. Unless other users are negligent, one user can’t remove directories belonging to
other users. The concept of ownership is discussed in Chapter 4.
How Files and Directories Are Created and Removed
A file (ordinary or directory) is associated with a name and a number, called the
inode number. When a file or directory is created, an entry comprising these two
parameters is made in the file’s parent directory. The entry is removed when the
file is deleted. Fig. 3.4 highlights the effect of mkdir and rmdir when creating and
removing the subdirectory progs in /home/romeo.
FIGURE 3.4
Filename Inode
Number
.
386444
..
417585
foo
499770

Directory Entry after mkdir and rmdir
Filename Inode
Number
.
386444
mkdir progs
..
417585
foo
499770
progs 162112

Filename Inode
Number
.
386444
rmdir progs
..
417585
foo
499770

We’ll discuss the significance of the entries . and .. that you’ll find in every
directory. In this chapter and the next, we’ll be monitoring this directory for changes
that are caused by some of the file-handling commands.

3.11 ls: Listing Files
The ls (list) command lists files—that is, their names. By default (i.e., when used without
arguments), it reads the current directory for the list. The default output could show the
filenames in multiple columns:
$ ls
08_packets.html
TOC.sh
calendar
cptodos.sh
dept.lst
emp.lst

das76205_Ch03_056-091.indd 68

helpdir
progs
usdsk06x
usdsk07x
usdsk08x

12/13/11 10:43 AM

69

Chapter 3: The File System

Viewed from top to bottom and then from left to right, the default output is ordered in
ASCII collating sequence (numbers first, uppercase, and then lowercase). However,
using certain options, the ordering sequence can be altered.
ls can also be used with one or more filenames to check whether a file is available:
$ ls calendar /bin/perl
calendar
/bin/perl: No such file or directory

calendar available
but not /bin/perl

When the Argument Is a Directory The behavior of ls changes when its argument is a directory. Rather than simply display the name of the directory (like it did for
calendar above), ls displays its contents:
$ ls helpdir
forms.hlp

graphics.hlp

reports.hlp

There are three files in the directory helpdir. But you can also make ls display
simply the name of the directory without listing its contents. Use the -d option:
ls -d helpdir.

Linux

ls can be configured to display filenames in different colors. Generally, executables
and directories are shown in separate colors. Run the alias command, and make sure
that ls is redefined on your system like this:
alias ls=’ls --color=tty’

If you don’t see this output, then simply run the alias command as shown above.
alias is an internal command of the Bash shell, and it’s likely that you would be
using this shell if you are using Linux.

3.11.1

ls Options
In this chapter, we discuss a few ls options from Table 3.1. The other options are taken
up in later chapters. We have already used one (-d) for suppressing a listing of the contents of a directory. On many systems, ls displays filenames in multiple columns by
default, but if that doesn’t happen on your system, use the -x option.
Identifying Directories and Executables (-F) The default output of ls doesn’t
identify directories or binary executables, but the -F option does. Combining it with -x
produces a multicolumnar output as well:
$ ls -Fx
08_packets.html
dept.lst
usdsk06x

das76205_Ch03_056-091.indd 69

Combining the -F and -x options

TOC.sh*
emp.lst
usdsk07x

calendar*
helpdir/
usdsk08x

cptodos.sh*
progs/
ux2nd06

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

70
TA B L E 3 . 1

Options to ls

Option

Description

-x
-F
-a
-R
-r
-l

Multicolumnar output
Marks executables with *, directories with /, and symbolic links with @
Shows all filenames beginning with a dot including . and ..
Lists files recursively
Sorts filenames in reverse order (ASCII collating sequence by default)
Long listing in ASCII collating sequence showing seven attributes of a
file (4.1)
Lists only dirname if dirname is a directory (4.1.1)
Sorts filenames by last modification time (4.10)
Sorts listing by last modification time (4.10)
Sorts filenames by last access time (4.10)
Sorts by ASCII collating sequence but listing shows last access time (4.10)
As above but sorted by last access time (4.10)
Displays inode number (4.6)

-d dirname
-t
-lt
-u
-lu
-lut
-i

Note the use of the * and / as type indicators. The * indicates that the file contains executable code and the / refers to a directory. There are two subdirectories here: helpdir
and progs.
Showing Hidden Files Also (-a) Filenames beginning with a dot have a special place
in the UNIX system. They are usually found in the home directory, but ls doesn’t show
them by default. Use the -a (all) option:
$ ls -axF
./
.exrc
.sh_history
08_packets.html*
.....

../
.kshrc
.shosts
TOC.sh*

.cshrc
.mozilla/
.vimrc
calendar*

.emacs
.profile
.xinitrc

There are several filenames here beginning with a dot. The file .profile contains a set
of instructions that are performed when a user logs in. It is discussed later. Another file,
.exrc (or .vimrc), contains a sequence of startup instructions for the vi editor. We’ll
also examine the significance of .shosts when we discuss the secure shell.
The first two files (. and ..) are special directories. Recall that we used the same
symbols in relative pathnames to represent the current and parent directories (3.8).
Whenever you create a subdirectory, these “invisible” directories are created automatically by the kernel. You can’t remove them, nor can you write into them. They help in
holding the file system together.

das76205_Ch03_056-091.indd 70

12/13/11 10:43 AM

71

Chapter 3: The File System

Recursive Listing (-R) The -R (recursive) option lists all files and subdirectories in
a directory tree. This traversal of the directory tree is done recursively until there are
no subdirectories left:
$ ls -xR
08_packets.html
dept.lst
usdsk06x

TOC.sh
emp.lst
usdsk07x

calendar
helpdir
usdsk08x

./helpdir:
forms.hlp

graphics.hlp

reports.hlp

./progs:
array.pl

cent2fah.pl

n2words.pl

cptodos.sh
progs
ux2nd06
Three files in helpdir

Four files in progs

name.pl

The list shows the filenames in three sections—the ones under the home directory
and those under the subdirectories helpdir and progs. Note the subdirectory naming
conventions followed; ./helpdir indicates that helpdir is a subdirectory under . (the
current directory).

Note

If ls displays a list of files when used with a single filename as argument, you can conclude
that the file is actually a directory. ls then shows the contents of the directory. The -d option
suppresses this behavior.

3.12 cp: Copying Files
We now take up the three essential commands that you can’t do without—cp (copy),
rm (remove), and mv (move or rename). Even though UNIX commands are generally
noninteractive, all three commands can also be made to run interactively.
Copies serve as good backups. The cp command copies one or more files or directory structures. The syntax requires at least two filenames (source and destination)
to be specified:
cp fork.c fork.c.bak

Even though we used simple filenames here, both source and destination can also
be pathnames. If the destination file (fork.c.bak) doesn’t exist, cp first creates it.
Otherwise, it simply overwrites the file without any warning. So check with ls whether
the destination file exists before you use cp.
The destination can also be a directory. The following example shows two ways
of copying a file to the progs directory:
cp fork.c progs/fork.c.bak
cp fork.c progs

das76205_Ch03_056-091.indd 71

fork.c copied to fork.c.bak under progs
fork.c retains its name under progs

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

72

cp is often used with the shorthand notation, . (dot), to signify the current directory as
the destination. The two commands that follow do the same thing; the second one uses
the ~ notation to signify juliet’s home directory:
cp /home/juliet/.profile .profile
cp ~juliet/.profile .

Destination is a file
Destination is the current directory

When cp is used to copy multiple files, the last filename must be a directory and must
already exist because cp won’t create it:
cp chap01 chap02 chap03 progs

progs must exist as a directory

If these files are already resident in progs, then they will be overwritten. The shell can
help in abbreviating this command line. You can use the metacharacter * as a suffix to
chap to match all of these filenames:
cp chap* progs

Copies all files beginning with chap

We’ll continue to use the * as a shorthand for multiple filenames. The metacharacters
related to filenames are discussed in Section 6.3. Can you do this job with ease using a
GUI program like the file manager?

Note

Caution

3.12.1

In the previous example, cp doesn’t look for a file named chap*. Before it runs, the shell expands
chap* to regenerate the command line arguments for cp to use.
cp will fail if the source is read-protected or the destination is write-protected. File permissions
are discussed in Section 4.2.

cp Options
Interactive Copying (-i) cp turns interactive when the -i (interactive) option is used
and the destination file also exists:
$ cp -i chap01 unit01
cp: overwrite unit01 (yes/no)? y

A y at this prompt overwrites the file; any other response leaves it uncopied. In
Section 8.4 we consider a technique by which cp can be made to behave in this
manner by default.
Copying Directory Structures (-R) The -R (recursive) option can be used to copy
an entire directory tree. This command copies all files and subdirectories in progs to
newprogs:
cp -R progs newprogs

newprogs must not exist

Attention! For this program to run in the way it is meant to, make sure that newprogs
doesn’t exist. cp -R will then create it as well as its associated subdirectories. Run the
command twice, and you’ll see different results!

das76205_Ch03_056-091.indd 72

12/13/11 10:43 AM

Chapter 3: The File System

73

3.13 mv: Renaming Files
Once you have used cp, you’ll feel comfortable with mv. This command renames a file
or directory. It can also move a group of files to a directory. This is how you rename
fork.txt to fork.c:
mv fork.txt fork.c

Creates or overwrites destination

You can move multiple files, but only to a directory. mv can also rename a directory:
mv fork1.c fork2.c fork3.c progs
mv progs c_progs

Or mv fork*.c progs
Directory renamed

As in cp -R, there’s a difference in behavior depending on whether c_progs exists or
not (see Self-Test). mv also supports a -i option which makes it behave interactively
(See Tip in Section 3.14.1).
Unlike its DOS counterpart (REN), mv can’t rename a group of files. In other words,
you can’t use mv *.avi *.xvid to change all .avi filename extensions to .xvid. UNIX
doesn’t have a command for this purpose, but Linux supports rename, which can perform
this change using the command rename .avi .xvid *.avi. However, the command
is not supported by POSIX.

3.14 rm: Deleting Files
Files tend to build up on disk and should be removed regularly to free disk space. The
rm command deletes files as well as directories. Here it deletes three files:
rm chap01 chap02 chap03

rm chap* could be dangerous to use!

rm is often used with a * to delete all files in a directory. Here it empties the directory
progs:
rm progs/*

rm can also clean up the current directory:
$ rm *
$ _

All files gone!

DOS users, beware! You won’t encounter the message All files in directory will
be deleted! It’s impossible for rm to behave in this manner because it doesn’t see the
* at all but only a list of filenames provided by the shell. Also, you need to be extremely
careful before deleting a file because a deleted file can’t be recovered. The * used here
is equivalent to *.* used in DOS.

Note

das76205_Ch03_056-091.indd 73

The * doesn’t match filenames beginning with a dot. So rm * leaves hidden files undeleted.
The technique of deleting such files is discussed in Section 6.3.3.

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

74
3.14.1

rm Options
Like cp and mv, rm -i also behaves interactively, so we’ll not discuss this option. Instead,
we examine two extremely risky options: -R and -f.
Recursive and Dangerous Deletion (-r or -R) The command rm * doesn’t remove
directories, but rm supports a -R (or -r) option to recursively delete an entire directory
tree. This command
rm -R *

Leaves out hidden files

deletes all files and subdirectories in the current directory. Note that the directories don’t
need to be empty for rm to remove them.
Forcing Removal (-f) rm doesn’t delete files that have the write permission removed.
Instead, it prompts for user confirmation, as is evident from this behavior on a Solaris
and Linux system:
rm: foo: override protection 444 (yes/no)?
rm: remove write-protected regular file `foo’?

Solaris
Linux

We’ll take up permissions and examine the significance of 444 in Chapter 4. rm will still
delete this file if you respond with a y. But the -f option overrides this minor protection
also. And when you combine the -r option with it, it could be the most dangerous thing
that you’ve ever done:
rm -rf *

Deletes everything in the current directory and below

If you don’t have a backup, then these files will be lost forever. Note that this command
will delete hidden files in all directories except the current directory.
Even though the cp, rm, and mv commands use simple syntaxes, you’ll often be
using them with pathnames—both absolute and relative. Table 3.2 shows how these
commands can handle a wide variety of arguments.
TA B L E 3 . 2

Usage of cp, rm, and mv Commands

Command line

Action

cp note ..
cp ../note .

Copies file note to the parent directory
Copies file note from the parent directory to the current
directory
Deletes file index in the bar directory placed at the same
hierarchical location as the current directory
Moves files foo1 and foo2 to the directory /foo1/foo2
Deletes complete directory structure of bar. Will delete
only bar if it is an ordinary file
Copies current directory tree to the directory bar under
the parent directory (bar must exist)
Moves all files from the parent directory to the current
directory

rm ../bar/index
mv foo1 foo2 /foo1/foo2
rm -r bar
cp -r . ../bar
mv ../* .

das76205_Ch03_056-091.indd 74

12/13/11 10:43 AM

75

Chapter 3: The File System

Caution

Tip

Make sure you are doing the right thing before you use rm *. Be doubly sure before you use
rm -rf *. The first command removes only ordinary files in the current directory. The second
one removes everything—files and directories alike. If the root user (the superuser) invokes
rm -rf * in the / directory, the entire UNIX system will be wiped out from the hard disk!
To protect your files from accidental overwriting or deletion you can redefine the three commands so that they always execute with the -i option. Aliases are taken up in Chapter 8, but
you can use one here. The syntax of the definition depends on the shell you use:
Korn and Bash
alias cp=”cp -i”
alias mv=”mv -i”
alias rm=”rm -i”

C Shell
alias cp “cp -i”
alias mv “mv -i”
alias rm “rm -i”

The Bourne shell doesn’t support aliases, where shell functions would be the choice. Shell
functions are discussed in Chapter 13.

How a Directory is Affected by cp, mv,and rm
cp, mv, and rm work by modifying the directory entry of the file they work on. As
shown in Fig. 3.5, cp adds an entry to the directory with the name of the file, and
the inode number that is allotted by the kernel. mv replaces the name of an existing
directory entry without disturbing its inode number. rm removes an entry from the
directory.
FIGURE 3.5

Directory Status after cp, mv, and rm

Filename Inode
Number
.
386444
..
417585
foo
499770

Filename
cp foo foo.bak

Inode
Number
.
386444
..
417585
bar
499770
foo.bak 509876

mv foo bar

Filename Inode
Number
.
386444
..
417585
bar
499770

das76205_Ch03_056-091.indd 75

Filename
rm foo.bak

Inode
Number
.
386444
..
417585
bar
499770
foo.bak 509876

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

76

This is a rather simplistic view, and is true only when source and destination
are in the same directory. When you “mv” a file to a directory that resides on a
separate hard disk, the file is actually moved. You’ll appreciate this better after you
have understood how multiple file systems create the illusion of a single file system
on your UNIX machine.
The action of rm also needs to be studied further. A file is not actually removed
by deleting its directory entry. There could be “similar” entries (ones having the
same inode number) for this file in this or another directory. We’ll examine this
directory table again when we take up file attributes in Chapter 4.

3.15 cat: Displaying and Concatenating Files
Most of the remaining commands in this chapter are concerned with content handling.
We begin with cat, the command that displays the contents of one or more files. It’s
suitable for displaying small files:
$ cat /etc/passwd
root:x:0:1:Super-User:/:/usr/bin/bash
daemon:x:1:1::/:
bin:x:2:2::/usr/bin:
sys:x:3:3::/:
adm:x:4:4:Admin:/var/adm:
lp:x:71:8:Line Printer Admin:/usr/spool/lp:
.......Remaining lines suppressed .......

This is how user information is structured. We’ll discuss the significance of each of the
seven fields in Chapter 19, but just note that the root user uses Bash as the shell on this
Solaris system. cat simply prints each byte in the file without any header and trailer
information.
The name “cat” expands to “concatenation,” which means it can concatenate
multiple files. cat’s output by default comes to the terminal, but we often save it in a
file using the shell’s > symbol:
cat foo.c foo1.c foo2.c > foo4.c

We don’t use cat to view executables because it produces junk. In Chapter 6, we’ll use
cat to even create a file to highlight an important feature of the shell.

3.16 more: The UNIX Pager
The man command internally uses more (less in Linux) to display its output a page at
a time. more today has replaced pg, the original pager of UNIX. Linux offers both more
and less (discussed in an aside at the end of this section). You need to use more rather
than cat to view large files:
more /etc/inetd.conf

das76205_Ch03_056-091.indd 76

Press q to exit

12/13/11 10:43 AM

77

Chapter 3: The File System

Apart from the first page, you also see at the bottom the filename and percentage of the
file that has been viewed:
--More--(17%)

Like mailx, more is used with its internal commands that don’t show up when you invoke them. q, the exit command, is an internal command. The AT&T and BSD versions
of this command differ widely in both usage and capabilities. The POSIX specification
is based on the BSD version. You have to try out the commands shown in Table 3.3 to
know whether they apply to your system. more has a fairly useful help screen too; hitting h invokes this screen.
Navigation You must have viewed a number of man pages by now, so you should be
familiar with these two navigation keys:
f or the spacebar
b

One page forward
One page back

Remember that the letters are not displayed on the screen. These navigation commands
and many others can be prefixed by a number called the repeat factor. This simply
repeats the command that many times. This means you can use 10f for scrolling forward
by 10 pages and 30b for scrolling back 30 pages. (vi also uses this feature; both more
and vi were developed at Berkeley.)
Repeating the Last Command (.) more has a repeat command, the dot (the
same command used by vi), which repeats the last command you used. If you scroll
TA B L E 3 . 3

das76205_Ch03_056-091.indd 77

Internal Commands of more and less

more

less

Action

Spacebar or f
20f
b
15b
[Enter]
/pat
n
. (a dot)
v
!cmd
q
h

Spacebar or f or z
b
j or [Enter]
k
p or 1G
G
/pat
n
?pat
v
!cmd
q
h

One page forward
20 pages forward
One page back
15 pages back
One line forward
One line back
Beginning of file
End of file
Searches forward for expression pat
Repeats search forward
Searches back for expression pat
Repeats last command
Starts up vi editor
Executes UNIX command cmd
Quit
Help

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

78

forward with 10f, you can scroll another 10 pages by simply pressing a dot. This is
a great convenience!
Searching for a Pattern You have seen the pattern search feature when using man.
Press a / and then the pattern:
/ftp[Enter]

Looks for ftp

You can repeat this search by pressing n as many times until you have scanned the entire
file. Move back with b (using a repeat factor, if necessary) to arrive at the first page.
Using more in a Pipeline We often use more to page the output of another command.
The ls output won’t fit on the screen if there are too many files, so the command has
to be used like this:
ls | more

No filename with more!

We have a pipeline here of two commands where the output of one is used as the input
of the other. Pipes are a feature of the shell, and the topic is taken up in Chapter 6.
less—The Standard Pager
Linux

Even though every Linux system offers more, its standard pager is ironically named
less. In many respects it is a superset of more. You’ll find vi features in less as well,
like the commands that permit one-line movement:
j

One line up

k

One line down

Unlike more, less can search for a pattern in the reverse direction also. The sequence
?ftp searches backwards for ftp. But less does have one serious limitation: unlike
more (which uses the .), it can’t repeat the last command.

3.17 pico: Rudimentary Text Editing
In Section 1.6.3, we used a workaround to create a file. We used echo with the > symbol
to create the file foo containing the string date. It’s time to learn a program that performs
the task of creating and editing a file. Programs that handle editing are appropriately
called editors, and every UNIX system has a number of them on offer. In this section
we consider a simple one, viz., the pico command, and in Chapter 5 we present a very
powerful one (the vi/vim editor).
The pico command is normally invoked with a filename. The file is created if it
doesn’t already exist:
pico foo

You’ll be presented a full screen (Fig. 3.6), where the topmost line and three lines at
the bottom are reserved for messages. The rest of the screen can be used for text input

das76205_Ch03_056-091.indd 78

12/13/11 10:43 AM

79

Chapter 3: The File System
Inserting Text in pico

FIGURE 3.6
UW PICO(tm)

4.10

File: foo

Modified

This is the pico editor.[Enter]
You need to know the commands for navigation.[Enter]
You also need to know the commands for cut-and-paste operations.[Enter]
Don't forget to save the buffer before you quit pico.[Enter]
There is a very useful help facility that is invoked by pressing ^G.

^G Get Help
^X Exit

^O WriteOut
^J Justify

^R Read File
^W Where is

^Y Prev Pg
^V Next Pg

^K Cut Text
^C Cur Pos
^U UnCut Text ^T To Spell

with the cursor initially positioned at the beginning of the first line. Observe the status
line at the top before you enter a few lines of text as shown, with each line followed by
[Enter]. The moment you have entered a character, the message Modified appears at
the top right. This indicates that you have made some change to a buffer (a temporary
work area) that has not been saved to disk yet.
The two lines at the bottom show the commonly used key sequences that you’ll
need for text editing. All navigation and editing functions in pico are carried out
using the Control key (shown as ^) along with a letter. For instance, [Ctrl-g] (shown
as ^G) invokes the help screen, which lists the complete set of internal commands
offered by this editor. Before we discuss the navigation and editing functions, let’s
save our work. Press [Ctrl-o] when a message pops up at the lower part of the screen
showing the default filename:
File Name to write : foo

You can change the filename if you want, but if you don’t want to, simply press the
[Enter] key. Your work has been saved. At this point you can quit the editor by pressing [Ctrl-x].
3.17.1

Navigation
You can move in the four directions either by using the cursor keys or the ones that follow:
[Ctrl-p]
[Ctrl-n]
[Ctrl-f]
[Ctrl-b]

One line up
One line down
One character forward
One character back

You can also move to the beginning or end of a line by using [Ctrl-a] and [Ctrl-e],
respectively. Movement by units of pages is also possible; use the help screen to find
out what the commands are.

das76205_Ch03_056-091.indd 79

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

80
3.17.2

Text Editing
Editing operations involve changing text in the buffer. While entering text, you’ll need
to erase text. Simply press either the [Delete] key or [Ctrl-d]. You’ll also need to move
text from one section of the buffer to another. pico doesn’t support copy-paste operations, but it allows you to cut text at one location and paste it at another location. For
this purpose, a block of text has to be selected. The beginning of the block is marked by
pressing [Ctrl-^] (shown as ^^ in the help screen). Now, use the right cursor or [Ctrl-f]
to select text which gets highlighted as the cursor is moved right. Once text has been
selected in this way, cut it by pressing this key sequence:
[Ctrl-k]

The text gets deleted. Move the cursor to the point where the deleted text has to be
pasted and then press
[Ctrl-u]

Shown as Uncut Text in the help screen.

The deleted text appears at the new location. In case you have to copy text, use a workaround: Cut the text normally, paste it first at the original location and then again at the
new location.
The pico editor is useful for kickstarting your UNIX learning experience, but
eventually as a programmer you’ll need to use a more powerful editor. The vi/vim
editor is presented in Chapter 5.

3.18 wc: Counting Lines, Words and Characters
The wc (word count) command counts lines, words, and characters. It takes one or more
filenames as its arguments and displays a four-columnar output. Let’s first “cat” a file:
$ cat infile
I am the wc command
I count characters, words and lines
With options I can also make a selective count

Now run wc without options to verify its own claim made above:
$ wc infile
3
20

103 infile

wc counts 3 lines, 20 words, and 103 characters. The filename has also been shown in
the fourth column. The meanings of these terms should be clear to you as they are used
throughout the book:
• A line is any group of characters not containing a newline.
• A word is a group of characters not containing a space, tab, or newline.
• A character is the smallest unit of information and includes a space, tab, and
newline.

das76205_Ch03_056-091.indd 80

12/13/11 10:43 AM

81

Chapter 3: The File System
wc offers three options to make a specific count. The -l option makes a line count:
$ wc -l infile
3 infile

Number of lines

The -w and -c options count words and characters, respectively. Like cat, wc doesn’t
work with only files; it also acts on a data stream. You’ll learn all about these streams
in Chapter 6.

3.19 lp: Printing a File
The printing system in UNIX requires a user to spool (line up) a job along with others
in a print queue. A separate program monitors this queue and then picks up each job in
turn for printing. The spooling facility in System V is provided by the lp (line printing)
and cancel commands. Linux uses the BSD system.
You must have your printer configured properly before you can use lp. The following lp command prints a single copy of the file rfc822.ps (a document containing
an Internet specification in the form of a Request For Comment):
$ lp rfc822.ps
request id is pr1-320 (1 file)
$ _

A Postscript file
Prompt returns immediately

lp notifies the request-id, a unique string that can later be accessed with other commands.
The job will be picked up from the queue and printed on the default printer. If the default is not defined or if there is more than one printer in the system, use the -d option:
lp -dlaser chap01.ps

Printer name is laser

You can notify the user with the -m (mail) option after the file has been printed. You can
also print multiple copies (-n):
lp -n3 -m chap01.ps

Prints three copies and mails user a message

Even though we used lp with filenames, this will not always be the case. You are aware
that the shell’s | symbol allows us to use ls | more. The same symbol also lets us
use ls | lp.
3.19.1

Other Commands in the lp Subsystem
The print queue is viewed with the lpstat (line printer status) command. By viewing
this list, you can use the cancel command to cancel any jobs submitted by you. cancel
uses the request-id or printer name as argument:
cancel laser
cancel pr1-320

das76205_Ch03_056-091.indd 81

Cancels current job on printer laser
Cancels job with request-id pr1-320

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

82

You can cancel only those jobs that you own (i.e., you have submitted yourself), but the
system administrator can cancel any job. cancel is effective only when a job remains
in the print queue. If it is already being printed, cancel can’t do a thing.
How UNIX Printers Work
Most UNIX printers are of the Postscript variety; i.e., they can properly print
files formatted in Postscript, like the files rfc822.ps and chap01.ps used in the
examples. (Postscript files are easily identified by the extension .ps.) When you
select Print from the File menu of any GUI program, the program converts the data
to Postscript, which then serves as input to the printer.
No such conversion, however, takes place when you use lp to print a text file
like /etc/passwd. If you have a text file to print, use a Postscript conversion utility
before you use lp. On Solaris, you can use the program
/usr/lib/lp/postscript/postprint before running lp.
Linux has a rich set of tools that convert text files to Postscript. Check whether
you have the programs a2ps or enscript on your system. Both eventually call up
lpr, the BSD printing program used by Linux.

Printing with lpr, lpq, and lprm
Linux

Berkeley devised its own system for printing that has been subsequently adopted by
many UNIX systems as well as Linux. This system uses the lpr command for printing.
The command normally doesn’t throw out the job number:
lpr /etc/group

As in System V, you can mail job completion, print a specific number of copies, and
direct output to a specific printer:
lpr -P hp4500 foo.ps
lpr -#3 foo.ps
lpr -m foo.ps

Prints on printer hp4500
Prints 3 copies
Mails message after completion

lpq displays the print queue showing the job numbers. Using one or more job numbers
as arguments to lprm, you can remove from the print queue only those jobs that are
owned by you:
lprm 31
lprm -

Removes job number 31
Removes all jobs owned by the user

3.20 od: Viewing Nonprintable Characters
Binary files contain nonprinting characters, and most UNIX commands don’t display
them properly. You can easily identify an executable file written in C by examining its
first few bytes, which is known as the magic number. These bytes are often characters
in the extended ASCII range and can’t be viewed with cat or more. We need to use od.

das76205_Ch03_056-091.indd 82

12/13/11 10:43 AM

83

Chapter 3: The File System

The od (octal dump) command displays the octal value of each character in its
input. When used with the -bc options, the output is quite readable. We’ll use od -bc to
look at the executable /bin/cat and /bin/ls. Since the output won’t fit on the screen,
we need to pipe it to more or less. We show below the first 16 characters of cat and
32 characters (2 lines) of ls:
$ od -bc /bin/cat | more
0000000 177 105 114 106
177 E
L
F
$ od -bc /bin/ls | less
0000000 177 105 114 106
177 E
L
F
0000020 002 000 003 000
002 \0 003 \0

001 002 001 000 000 000 000 000 000 000 000 000
001 002 001 \0 \0 \0 \0 \0 \0 \0 \0 \0
001
001
001
001

001
001
000
\0

001
001
000
\0

000 000 000 000 000 000 000 000 000
\0 \0 \0 \0 \0 \0 \0 \0 \0
000 060 221 004 010 064 000 000 000
\0
0 221 004 \b
4 \0 \0 \0

Each line displays 16 bytes of data in octal, preceded by the offset (position) in the file
of the first byte in the line. The first character has the ASCII octal value 177, and the
next three comprise the string ELF. All C executables have the same first four characters. The second line shows the text representation of each character wherever possible.
You can try a similar exercise with tar archives and “gzipped” files after you have
completed this chapter. od also displays escape sequences like \r and \n, and we’ll see
them when we use the command again for examining DOS files.

3.21 dos2unix, unix2dos, and Tofrodos:
Converting Between DOS and UNIX
You’ll sometimes need to move files between Windows and UNIX systems. Windows files
use the same format as DOS, where the end of line is signified by two characters—CR
(\r) and LF (\n). UNIX files, on the other hand, use only LF. Here are two lines from
a DOS file, foo, viewed on a UNIX system with the vi editor:
Line 1^M
Line 2^M

The [Ctrl-m] character at end

There’s a ^M ([Ctrl-m]) representing the CR sequence at the end of each line. An octal
dump confirms this:
$ od -bc foo
0000000 114 151 156 145 040 061 015 012 114 151 156 145 040 062 015 012
L
i n
e
1 \r \n
L i
n
e
2 \r \n

Conversion of this file to UNIX is just a simple matter of removing the \r. Some
UNIX systems feature two utilities—dos2unix and unix2dos—for converting files
between DOS and UNIX. The behavior of these commands varies across systems,

das76205_Ch03_056-091.indd 83

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

84

so you need the help of man to determine which of the following actually works on
your system:
dos2unix
dos2unix
dos2unix
dos2unix

foo foo.unix
foo > foo.unix
foo
foo foo

Output written to foo.unix—Solaris
Taking shell’s help
Output written back to foo—Linux
Same as above—Solaris

When you use od again, you’ll find that the CR character is gone:
$ od -bc foo.unix
0000000 114 151 156 145 040 061 012 114 151 156 145 040 062 012
L
i n
e
1 \n
L
i n
e
2 \n

unix2dos inserts CR before every LF, and thus increases the file size by the number of
lines in the file. The syntactical form that works for dos2unix also works for unix2dos.
You could also consider the Tofrodos package that contains two commands, todos
and fromdos. The default output for both is written back to the same file:
todos foo
fromdos foo

Converts to DOS
Converts to UNIX

Irrespective of what the names suggest, either command can perform both tasks using the -d option. Thus, todos -d foo is the same as fromdos foo, and vice versa.
Use the -h option to know about the other options. One of them (-b) lets you save the
original as a .bak file.

Note

Caution

If you view the UNIX file foo.unix on Windows using Notepad but without performing the
conversion, you’ll see a single line, Line 1Line2. In fact, whenever you see a single line on a
Windows machine that should have been multiple lines, satisfy yourself that you are viewing
an unconverted UNIX file.
Never perform this conversion on a binary file. If you have downloaded a Windows program
(say, a .EXE file) on a UNIX machine, the file must be transferred to the Windows machine
without conversion. Otherwise, the program simply won’t execute.

3.22 tar: The Archival Program
For sending a group of files to someone either by FTP (the file transfer protocol) or email,
it helps to combine them into a single file called an archive. The tar (tape archiver)
command is an archiver which we consider briefly here and in detail in Chapter 19. It
supports these key options; only one option can be used at a time:
-c
-x
-t

das76205_Ch03_056-091.indd 84

Creates an archive
Extracts files from archive
Displays files in archive

12/13/11 10:43 AM

85

Chapter 3: The File System

In addition, we’ll frequently use two options: -f for specifying the name of the archive
and -v to display the progress. This is how we create a file archive, archive.tar, from
two uncompressed files:
$ tar -cvf archive.tar libc.html User_Guide.ps
a libc.html 3785K
a User_Guide.ps 364K

-v (verbose) displays list
a indicates append

By convention, we use the .tar extension, so you’ll remember to use the same tar
command for extraction. Move this .tar file to another directory and then use the -x
option for extracting the two files:
$ tar -xvf archive.tar
x libc.html, 3875302 bytes, 7569 tape blocks
x User_Guide.ps, 372267 bytes, 728 tape blocks

Extracts files
x indicates extract

You’ll now find the two files in the current directory. tar is most useful for archiving
one or more directory trees.
To view the contents of the archive, use the -t (table of contents) option. It doesn’t
extract files, but simply shows their attributes in a form that you’ll see more often later:
$ tar -tvf archive.tar
-rw-r--r-- 102/10 3875302 Aug 24 19:49 2002 libc.html
-rw-r--r-- 102/10 372267 Aug 24 19:48 2002 User_Guide.ps

You’ll understand the significance of these columns after you have learned to interpret
the ls -l output. But you can at least see the individual file size (third column) and the
filename (last column) in this output.

3.23 gzip: The Compression Program
Eventually you’ll encounter compressed files on the Internet or need to compress one
yourself before sending it as an email attachment. Your UNIX system may have one or
more of these compression programs: gzip, bzip2, and zip. The degree of compression
depends on the type of file, its size, and the compression program used. In this section
we take up gzip and its decompression sibling, gunzip.
gzip provides the extension .gz to the compressed filename and removes the
original file. This command compresses an HTML file:
gzip libc.html

Replaces with libc.html.gz

To see the amount of compression achieved, use the -l option:
$ gzip -l libc.html.gz
compressed uncompr. ratio uncompressed_name
788096
3875302 79.6% libc.html

das76205_Ch03_056-091.indd 85

.gz not necessary

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

86

Uncompressing a “gzipped” File (-d) To restore the original and uncompressed
file, you have two options: use either gzip -d or gunzip:
gunzip libc.html.gz
gzip -d libc.html.gz

Retrieves lib.html
Same

You’ll have to understand why two commands have been offered to do the same job
when one of them would have sufficed. Are gzip and gunzip one and the same file?
Is the same true for todos and fromdos? This question is related to file attributes, and
we discuss file attributes in Chapter 4.
To view compressed plain text files, you really don’t need to “gunzip” (decompress) them.
Use the gzcat and gzmore (or zcat and zmore) commands if they are available on your system.
Tip

Using with tar An additional layer of compression helps bring down the file size,
the reason why gzip is often used with tar for creating a compressed archive. Here we
“gzip” the file archive.tar that was created in Section 3.21 with tar:
gzip archive.tar

Archived and compressed

This creates a “tar-gzipped” file, archive.tar.gz. A shorthand notation of the .tar.gz
extension is often seen: .tgz. This file can now be sent out by FTP or as an email attachment to someone. To extract the files from this compressed archive, we simply have to
reverse the procedure: use gunzip to decompress the archive and then run tar:
gunzip archive.tar.gz
tar -xvf archive.tar

Retrieves archive.tar
Extracts libc.html and User_Guide.ps

A great deal of open-source UNIX and Linux software is available as .tar.gz or .tgz
files on the Internet. To be able to extract files from this archive, the recipient needs to
have both tar and gunzip (or gzip) at her end.

Linux

tar and gzip are so often used together that GNU tar has a -z option that compresses
and archives together (and decompresses and extracts together). This dual activity is
reflected in the following commands:
tar -cvzf archive.tar.gz libc.html User_Guide.ps
tar -xvzf archive.tar.gz

Compresses also
Decompresses also

Note that whether you should use the -z option with -x depends on whether the archive
was compressed in the first place with -c. The archive’s extension (.tar.gz or .tgz)
should provide this hint, but that’s no guarantee. A wrong extension could have been
provided by the user at the time of archiving. If you are working on a DOS system,
then you can use the later versions of the WinZip or WinRAR programs which can
also handle these files. Mac OS X users can use Stuffit.

das76205_Ch03_056-091.indd 86

12/13/11 10:43 AM

Chapter 3: The File System

Note

87

For some years, gzip reigned as the most favored compression agent. Today we have a better
agent in bzip2 (and bunzip2). bzip2 is slower than gzip and creates .bz2 files. bzip2 options
are modeled on gzip, so if you know gzip you also know bzip2. GNU tar also supports compression with bzip2 in the --bzip2 option. Provide the extension .tar.bz2 to the compressed
archive so that the person at the other end knows how to handle it.

3.24 zip: The Compression and Archival Program
The popular PKZIP and PKUNZIP programs are now available as zip and unzip on
UNIX. zip combines the compressing function of gzip with the archival function of tar.
So instead of using two commands to compress an archive (say, a directory structure),
you can use only one—zip.
zip requires the first argument to be the compressed filename; the remaining
arguments are interpreted as files and directories to be compressed. A previous archival
and subsequent compression in two previous examples could have been achieved with
zip in the following way:
$ zip archive.zip libc.html User_Guide.ps
adding: libc.html (deflated 80%)
adding: User_Guide.ps (deflated 66%)

The unusual feature of this command is that it doesn’t overwrite an existing compressed
file. If archive.zip exists, files will either be updated or appended to the archive depending on whether they already exist in the archive.
Recursive Compression (-r) For recursive behavior, zip uses the -r option. It descends the tree structure in the same way tar does except that it also compresses files.
You can easily compress your home directory in this way:
cd ; zip -r sumit_home.zip .

cd is same as cd $HOME

Using unzip zip files are decompressed with unzip. unzip does a noninteractive
restoration if it doesn’t find the same files on disk:
$ unzip archive.zip
Archive: archive.zip
inflating: libc.html
inflating: User_Guide.ps

But if the uncompressed file exists on disk, then unzip makes sure that it’s doing the
right thing by seeking user confirmation:
replace libc.html? [y]es, [n]o, [A]ll, [N]one, [r]ename: y

You can respond with y or n. You can also rename the file (r) to prevent overwriting, or
direct unzip to perform the decompression on the remaining files noninteractively (A).

das76205_Ch03_056-091.indd 87

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

88

Viewing the Archive (-v) You can view the compressed archive with the -v option.
The list shows both the compressed and uncompressed size of each file in the archive
along with the percentage of compression achieved:
$ unzip -v archive.zip
Archive: archive.zip
Length Method Size
------ ------ ---3875302 Defl:N 788068
372267 Defl:N 128309
----------4247569
916377

3.25

Ratio
Date
-------80% 08-24-02
66%
08-24-02
--78%

Time
---19:49
19:48

CRC-32
-----fae93ded
7839e6b3

Name
---libc.html
User_Guide.ps
------2 files

Other Ways of Using These Commands
The commands discussed in this chapter don’t always take input from files. Some commands (like more and lp) use, as alternate sources of input, the keyboard or the output
of another command. Most of the other commands (like wc, cat, od, gzip, and tar) can
also send output to a file or serve as input to another command. Some examples in this
chapter (and previous ones) have shown this to be possible with the > and | symbols.
The discussion of these techniques is taken up in Chapter 6.

SUMMARY
We considered three types of files—ordinary, directory, and device. A directory maintains
the inode number and name for each file. The kernel uses the attributes of a device file
to operate the device. File attributes are maintained in the inode.
A filename is restricted to 255 characters and can use practically any character.
Executable files don’t need any specific extensions.
UNIX supports a hierarchical file system where the topmost directory is called
root. An absolute pathname begins with a / and denotes the file’s location with respect
to root. A relative pathname uses the symbols . and .. to represent the file’s location
relative to the current and parent directory, respectively.
pwd tells you the current directory, and cd is used to change it or to switch to the
home directory. This directory is set in /etc/passwd and is available in the shell variable HOME. A file foo in the home directory is often referred to as $HOME/foo or ~/foo.
mkdir and rmdir are used to create or remove directories. To remove a directory
bar with rmdir, bar must be empty and you must be positioned above bar.
By default, ls displays filenames in ASCII collating sequence (numbers, uppercase,
lowercase). It can also display hidden filenames beginning with a dot (-a). When used
with a directory name as argument, ls displays the filenames in the directory.
You can copy files with cp, remove them with rm, and rename them with mv.
All of them can be used interactively (-i), and the first two can be used to work on a
complete directory tree (-r or -R) i.e., recursively. rm -r can remove a directory tree
even if is not empty.

das76205_Ch03_056-091.indd 88

12/13/11 10:43 AM

Chapter 3: The File System

89

cat and more are used to display the contents of a file. more supports a number
of internal commands that enable paging and searching for a pattern. Linux offers less
as a superior pager.
Rudimentary editing tasks can be performed with the pico editor. The program
uses the Control keys for all navigation and editing operations. You can perform cutand-paste operations using the keys [Ctrl-k] and [Ctrl-u], respectively.
lp submits a job for printing which is actually carried out by a separate program.
Linux and many UNIX systems use the lpr command for printing. Both can be directly
used to print Postscript documents.
wc counts the number of lines, words, and characters. od displays the octal value
of each character and is used to display invisible characters.
The dos2unix and unix2dos commands convert files between DOS and UNIX.
DOS files use CR-LF as the line terminator, while UNIX uses only LF. The todos and
fromdos commands also perform the same functions.
gzip and gunzip compresses and decompresses individual files (extension: .gz).
tar can archive a directory tree and is often used with gzip to create compressed archives
(extension: .tar.gz or .tgz). zip and unzip use .zip files. zip alone can create a
compressed archive from directory structures (-r). bzip2 is better than zip and unzip
(extension: .bz2).

SELF-TEST
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
3.10
3.11
3.12
3.13
3.14
3.15

das76205_Ch03_056-091.indd 89

How long can a UNIX filename be? What characters can’t be used in a filename?
State two reasons for not having a filename beginning with a hyphen.
Name the two types of ordinary files, and explain the difference between them.
Provide three examples of each type of file.
Can the files note and Note coexist in the same directory?
Frame cd commands to change from (i) /var/spool/lp/admins to
/var/spool/mail, (ii) /usr/include/sys to /usr.
Switch to the root directory with cd, and then run cd .. followed by pwd. What
do you notice?
Explain the significance of these two commands: ls .. ; ls -d ...
Can you execute any command in /sbin and /usr/sbin by using the absolute
pathname?
If the file /bin/echo exists on your system, are the commands echo and
/bin/echo equivalent?
Look up the man pages of mkdir to find out the easiest way of creating this
directory structure: share/man/cat1.
If mkdir test fails, what could be the possible reasons?
How do you run ls to (i) mark directories and executables separately, (ii) also
display hidden files?
What will cat foo foo foo display?
A file contains nonprintable characters. How do you view them?
How will you copy a directory structure bar1 to bar2? Does it make any difference if bar2 exists?

12/13/11 10:43 AM

Your UNIX/Linux: The Ultimate Guide

90
3.16
3.17
3.18
3.19
3.20
3.21

Assuming that bar is a directory, explain what the command rm -rf bar does.
How is the command different from rmdir bar?
How do you print the file /etc/passwd on the printer named laser on System V
(i) to generate three copies, (ii) and know that the file has been printed?
How will you find out the ASCII octal values of the numerals and letters?
Run the wc command with two or more filenames as arguments. What do you see?
How can you abort an editing session with pico without saving the changes?
How can you join multiple lines in pico into a single line without using cut-andpaste operations?

EXERCISES
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9

3.10
3.11
3.12
3.13

3.14
3.15
3.16
3.17

das76205_Ch03_056-091.indd 90

Describe the contents of a directory, explaining the mechanism by which its entries
are updated by cp, mv, and rm. Why is the size of a directory usually small?
How does the device file help in accessing the device?
Which of these commands will work? Explain with reasons: (i) mkdir a/b/c,
(ii) mkdir a a/b, (iii) rmdir a/b/c, (iv) rmdir a a/b, (v) mkdir /bin/foo.
The command rmdir c_progs failed. State three possible reasons.
Using echo, try creating a file containing (i) one, (ii) two, and (iii) three dots.
What do you conclude?
The command rmdir bar fails with the message that the directory is not empty.
On running ls bar, no files are displayed. Why did the rmdir command fail?
How does the command mv bar1 bar2 behave, where both bar1 and bar2 are
directories, when (i) bar2 exists, (ii) bar2 doesn’t exist?
Explain the difference between the commands cd ~charlie and cd ~/charlie.
Is it possible for both commands to work?
charlie uses /usr/charlie as his home directory, and many of his scripts refer
to the pathname /usr/charlie/html. Later, the home directory is changed to
/home/charlie, thus breaking all his scripts. How could charlie have avoided
this problem?
Why do we sometimes run a command like this—./update.sh instead of
update.sh?
What is the sort order prescribed by the ASCII collating sequence?
The commands ls bar and ls -d bar display the same output—the string bar.
This can happen in two ways. Explain.
Assuming that you are positioned in the directory /home/romeo, what are
these commands presumed to do, and explain whether they will work at all:
(i) cd ../.., (ii) mkdir ../bin, (iii) rmdir .., (iv) ls ...
Explain what the following commands do: (i) cd, (ii) cd $HOME, (iii) cd ~.
The command cp hosts backup/hosts.bak didn’t work even though all files
exist. Name three possible reasons.
You have a directory structure $HOME/a/a/b/c where the first a is empty. How
do you remove it and move the lower directories up?
Explain what the following commands do: (i) rm *, (ii) rm -i *, (iii) rm -rf *.

12/13/11 10:43 AM

Chapter 3: The File System
3.18
3.19
3.20

3.21
3.22
3.23
3.24

3.25

3.26
3.27

3.28
3.29
3.30

das76205_Ch03_056-091.indd 91

91

What is the significance of these commands? (i) mv $HOME/include .,
(ii) cp -r bar1 bar2, (iii) mv * ../bin.
Will the command cp foo bar work if (i) foo is an ordinary file and bar is a
directory, (ii) both foo and bar are directories?
Explain the significance of the repeat factor used in more. How do you search
for the pattern include in a file and repeat the search? What is the difference
between this repeat command and the dot command?
Look up the man page for the file command, and then use it on all files in the
/dev directory. Can you group these files into two categories?
How do DOS and UNIX text files differ? Name the utilities that convert files
between these two formats.
Run the script command, and then issue a few commands before you run exit.
What do you see when you run cat -v typescript?
Run the tty command, and note the device name of your terminal. Now use this
device name (say, /dev/pts/6) in the command cp /etc/passwd /dev/pts/6.
What do you observe?
How do you use tar to add two files, foo.html and bar.html, to an archive,
archive.tar, and then compress the archive? How will you reverse the entire
process and extract the files in their original uncompressed form?
Name three advantages zip has over gzip.
How do you send a complete directory structure to someone by email using
(i) tar, (ii) zip? How does the recipient handle it? Which method is superior
and why? Does gzip help in any way?
What is meant by recursive behavior of a command? Name four commands,
along with a suitable example of each, that can operate recursively.
There are shortcuts available for many pico commands that use the function
keys. Use the help screen to find out the function keys that cut and paste text.
Use the pico help screen to find out the key sequence that places the contents
of another file in the file being edited.

12/13/11 10:43 AM

CHAPTER

4

File Attributes

I

n Chapter 3, you created directories, navigated the file system, and copied,
moved, and removed files without any problem. In real life, however, matters
may not be so rosy. You may have problems when handling a file or directory. Your file
may be modified or even deleted by others. A restoration from a backup may be unable
to write to your directory. You must know why these problems occur and how to prevent
and rectify them.
The UNIX file system lets users access files that don’t belong to them—without
infringing on security. A file also has a number of attributes that are changeable by certain well-defined rules. In this chapter, we’ll use the ls command in all possible ways
to display these attributes. We’ll also use other commands to change these attributes.
Finally, we’ll discuss find—one of the most versatile attribute handling tools of the
UNIX system.

Objectives
• Learn the significance of the seven fields of the ls -l output (listing).
• Use chmod to change file permissions in a relative and absolute manner.
• Understand the significance of directory permissions and how they ultimately impact
a file’s access rights.
• Understand the concept of the user mask and how umask changes the default file and
directory permissions.
• Become familiar with file systems and the inode as a file identifier.
• Create hard links to a file with ln.
• Learn the limitations of hard links and how they are overcome by symbolic links.
• Know the importance of ownership and group ownership of a file and how they
affect security.
• Use chown and chgrp to change the owner and group owner of files on BSD and
AT&T systems.
• Locate files by matching one or more file attributes with find.

4.1 ls Revisited (-l): Listing File Attributes
File attributes are stored in the inode, a structure that is maintained in a separate area
of the hard disk. Before we examine the contents of the inode, let’s first have a look
92

das76205_Ch04_092-121.indd 92

12/13/11 10:44 AM

93

Chapter 4: File Attributes
FIGURE 4.1
$ ls -l
total 24
-r--r--r--rw-rw-rw-rwxr-xr-x
-rwxr-xr-drwxr-xr-x
lrwxrwxrwx
-r--r--r--rwxr-xr-drwxrwxr-x

1
1
1
2
2
1
1
2
2

Listing of Files with ls -l

256
romeo
root
romeo
romeo
romeo
romeo
romeo
romeo

105
metal
root
metal
metal
metal
metal
metal
metal

13921
473
6496
163
512
17
268
163
512

Jul
Jul
Aug
Jul
Aug
Aug
Jul
Jul
Aug

26
13
10
13
10
11
13
13
10

2001
21:36
10:20
21:36
10:42
00:49
21:36
21:36
10:45

987
CallByRef.java
a.out
backup.sh
c_progs
hex.c -> c_progs/hexdump.c
prime.c
restore.sh
shell_scripts

at some of the major attributes which are listed by the ls -l command. The output in
UNIX lingo is often referred to as the listing, and a typical listing is shown in Fig. 4.1.
The list shows seven labeled fields in nine columns with the filenames ordered
in ASCII collating sequence. Each field here represents a file attribute, and all of these
attributes (except the filename) are stored in the inode. We’ll discuss most of these
attributes in detail in this chapter, but let’s understand their significance first.
Type and Permissions The first column of the first field shows the file type. Here
we see three possible values—a - (ordinary file), d (directory), or l (symbolic link).
Most files here are ordinary files, but c_progs and shell_scripts are directories.
We’ll discuss the symbolic link later. The remaining nine characters form a string of
permissions which can take the values r, w, x, and -.
Links The second field indicates the number of links associated with the file. UNIX
lets a file have multiple names, and each name is interpreted as a link. Directories have
a link count of at least two, but here two ordinary files (backup.sh and restore.sh)
also have two links each. Even though they have the same file size (163 bytes), that
doesn’t confirm that there’s only a single file out there.
Ownership and Group Ownership Every file has an owner. The third field shows
romeo as the owner of most of the files. A user also belongs to a group, and the fourth
field shows metal as the group owner of most of the files. The owner can tamper with
a file in every possible way—a privilege that is also available to the root user. We’ll
discuss how two files having different ownership and group ownership (987 and a.out)
have crept into this directory.
Size The fifth field shows the file size in bytes. This actually reflects the character
count and not the disk space consumption of the file. The kernel allocates space in
blocks of 1024 bytes or more, so even though backup.sh contains 163 bytes, it could

das76205_Ch04_092-121.indd 93

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

94

occupy 1024 bytes on this system. The two directories show smaller file sizes, but that
is to be expected because the size of a directory depends on the number of filenames it
contains—whatever the size of the files themselves.
Last Modification Time The sixth field displays the last modification time in three
columns—a time stamp that is stored to the nearest second. The file named 987 shows
the year; the year is displayed if more than a year has elapsed since it was last modified
(six months in Linux). You’ll often need to run automated tools that make decisions
based on a file’s modification time. This column shows two other time stamps when ls
is used with certain options.
Filename The last field displays the filename, which can be up to 255 characters long.
If you would like to see an important file at the top of the listing, then choose its name
in uppercase—at least, its first letter.
The entire list is preceded by the words total 24; a total of 24 blocks are occupied
by these files in the disk. There are other file attributes (like the inode number), and
sometimes we combine the -l option with other options for displaying other attributes
or ordering the list in a different sequence. We’ll now examine each of these attributes
and learn to change them.
4.1.1

Listing Directory Attributes (-ld)
We can use ls -l with filenames as arguments for a selective listing. But since ls bar
lists the contents of bar if it is a directory, we need to combine the -l and -d options
to force the listing:
$ ls -ld c_progs shell_scripts
drwxr-xr-x 2 romeo
metal
drwxrwxr-x 2 romeo
metal

512 Aug 10 10:42 c_progs
512 Aug 10 10:45 shell_scripts

While we maintain that a directory is also a file, the significance of its permissions, link
count, and size differ with ordinary files.
To see the attributes of a directory bar rather than the filenames it contains, use ls -ld bar.
Note that simply using ls -ld won’t show the listing of all subdirectories in the current directory. Strange though it may seem, ls has no option to list only directories!

Note

4.2 File Permissions
UNIX has a simple and well-defined system of assigning permissions to files. Observe
from the listing in Fig. 4.1 that permissions can vary a great deal between files. Let’s
examine the permissions of backup.sh:
-rwxr-xr--

2 romeo

metal

163 Jul 13 21:36 backup.sh

UNIX follows a three-tiered file protection system that determines a file’s access rights.
To understand how this system works, let’s break up the permissions string of this file

das76205_Ch04_092-121.indd 94

12/13/11 10:44 AM

95

Chapter 4: File Attributes
FIGURE 4.2

Structure of a File’s Permissions String
Owner’s permissions

-

r

w

x

Type of file

Others’ permissions
r

-

x

r

-

-

Group’s permissions

into three groups as shown in Fig. 4.2. The initial - (in the first column) signifies an
ordinary file and is left out of the permissions string.
Each group here represents a category. There are three categories representing the
user (owner), group owner, and others. Each category contains three slots representing
the read, write, and execute permissions of the file. r indicates read permission, which
means cat can display the file. w indicates write permission; you can edit such a file
with an editor. x indicates execute permission; the file can be executed as a program.
The - shows the absence of the corresponding permission.
The first category (rwx) shows the presence of all permissions. The file is readable, writable, and executable by the owner of the file. Identifying the owner is easy;
the third field shows romeo as the owner. You have to log in with the username romeo
for these privileges to apply to you.
In a similar manner, the second category (r-x) indicates the absence of write
permission for the group owner of the file. This group owner is metal as shown in the
fourth field. The third category (r--) applies to others (neither owner nor group owner).
This category is often referred to as the world. This file is world-readable, but others
can’t write or execute it.
A file or directory is created with a default set of permissions that is determined
by a simple setting (called umask), which we’ll discuss later. Different systems have
different umask settings, but to make sure that you also obtain the same initial permissions, use umask in this manner before creating a file, date.sh (a shell script), containing the string date:
$ umask 022
$ echo date > date.sh ; ls -l date.sh
-rw-r--r-- 1 romeo
metal
5 Aug 16 16:05 date.sh

All users have read permission, only the owner has write permission, but the file is not
executable by anyone:
$ ./date.sh
bash: ./date.sh: Permission denied

Preferred way to run shell scripts

How then does one execute such a file? Just change its permissions with the chmod
(change mode) command. With this command, you can set different permissions for the
three categories of users—owner, group, and others. It’s important that you understand

das76205_Ch04_092-121.indd 95

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

96

them because a little learning here can be a dangerous thing. A faulty file permission is
a sure recipe for disaster.
The group permissions here don’t apply to romeo (the owner) even if romeo belongs to the
metal group. The owner has its own set of permissions that override the group owner’s permissions. However, when romeo renounces the ownership of the file, the group permissions
then apply to him.

Note

4.3 chmod: Changing File Permissions
Before we take up chmod, let’s decide to change a habit. Henceforth, we’ll refer to the
owner as user because that’s how the chmod command (which changes file permissions)
refers to the owner. In this section, whenever we use the term user, we’ll actually mean
owner.
We’ll now use chmod to change a file’s permissions. The command uses the following syntax:
chmod [-R] mode file ...

POSIX specifies only a single option (-R). The mode can be represented in two ways:
• In a relative manner by specifying the changes to the current permissions.
• In an absolute manner by specifying the final permissions.
We’ll consider both ways of using chmod, but just remember that only the owner of this
file (romeo) can change these permissions.
4.3.1

Relative Permissions
When changing permissions in a relative manner, chmod only changes the permissions
specified in mode and leaves the other permissions unchanged. The structure of a chmod
command is shown in Fig. 4.3. The mode as used in the syntax contains three components:
• User category (user, group, others)
• The operation to be performed (assign or remove a permission)
• The type of permission (read, write, execute)

FIGURE 4.3

Structure of a chmod Command
Operation
Command
chmod

File
u

Category

das76205_Ch04_092-121.indd 96

+

x

note

Permission

12/13/11 10:44 AM

97

Chapter 4: File Attributes
TA B L E 4 . 1

Abbreviations Used by chmod

Category

Operation

Permission

u—User

+—Assigns permission

r—Read permission

g—Group
o—Others
a—All (ugo)

-—Removes permission
=—Assigns absolute permission

w—Write permission
x—Execute permission

To make the file date.sh executable, frame a suitable expression by using appropriate
characters from each of the three columns of Table 4.1. We need to assign (+) execute
permission (x) to the user (u). The expression required is u+x:
$ chmod u+x date.sh
$ ls -l date.sh
-rwxr--r-- 1 romeo
metal
$ ./date.sh
Sat Aug 16 16:16:12 GMT 2003

Execute permission for user

5 Aug 16 16:05 date.sh
Now execute the shell script

Permissions are removed with the - operator. This is how we revert to the original
permissions:
$ chmod u-x date.sh ; ls -l date.sh
-rw-r--r-- 1 romeo
metal

5 Aug 16 16:05 date.sh

The expression can comprise multiple categories. This is how we assign execute permission to all:
$ chmod ugo+x date.sh ; ls -l date.sh
-rwxr-xr-x 1 romeo
metal
5 Aug 16 16:05 date.sh

The synonym a is available for ugo, so ugo+x is the same as a+x (or even +x). We can
also assign multiple permissions:
$ chmod go-rx date.sh ; ls -l date.sh
-rwx------ 1 romeo
metal
5 Aug 16 16:05 date.sh

How do we revert now to the original permissions? We need to remove the execute permission from user and assign read permission to the other two categories. This requires
two expressions, and using a comma as a delimiter between them, we can use a single
invocation of chmod:
$ chmod u-x,go+r date.sh ; ls -l date.sh
-rw-r--r-- 1 romeo
metal
5 Aug 16 16:05 date.sh

However, this isn’t convenient; we had to use a complex expression to revert to the
default permissions. Whatever the current permissions are, we should be able to assign
permissions in an absolute manner. chmod can do that too. (Note that some versions of
chmod don’t accept the comma.)

das76205_Ch04_092-121.indd 97

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

98
4.3.2

Absolute Assignment
The = operator can perform a limited form of absolute assignment. It assigns only the
specified permissions and removes other permissions. Thus, if a file is to be made readonly to all, we can simply use one of these three forms:
chmod ugo=r date.sh
chmod a=r date.sh
chmod =r date.sh

This technique has its limitations; you can’t just set all nine permission bits explicitly.
Absolute assignment is actually done with octal numbers. You may or may not be familiar
with this numbering system, so a discussion would be in order.
Octal numbers use the base 8, and octal digits have the values 0 to 7. This means
that a set of three bits can represent one octal digit. If we represent the permissions of
each category by one octal digit, then this is how the permissions can be represented:
• Read permission—4 (Octal 100)
• Write permission—2 (Octal 010)
• Execute permission—1 (Octal 001)
For each category we add up the numbers. For instance, 6 represents read and write
permissions, and 7 represents all permissions as can easily be understood from the following table:
Binary

Octal

Permissions

Significance

000

0

---

No permissions

001

1

--x

Executable only

010

2

-w-

Writable only

011

3

-wx

Writable and executable

100

4

r--

Readable only

101

5

r-x

Readable and executable

110

6

rw-

Readable and writable

111

7

rwx

Readable, writable, and executable

We have three categories and three permissions for each category, so three octal digits
can describe a file’s permissions completely. The most significant digit represents user,
and the least significant one represents others. chmod can use this three-digit string as
the expression.
The default file permissions on our system are rw-r--r--. This is octal 644, so
let’s use it with chmod:
$ chmod 644 date.sh ; ls -l date.sh
-rw-r--r-- 1 romeo
metal

das76205_Ch04_092-121.indd 98

5 Aug 16 16:05 date.sh

12/13/11 10:44 AM

99

Chapter 4: File Attributes

Some network applications store the password in a configuration file in the home
directory. This file must be made unreadable to group and others. For this you need the
expression 600:
$ cd ; chmod 600 .netrc ; ls -l .netrc
-rw------- 1 romeo
metal
50 Aug 16 16:50 .netrc

It’s obvious that 000 indicates the absence of all permissions, and 777 signifies the presence of all permissions for all categories. But can we delete a file with permissions 000?
Yes, we can. Can we prevent a file with permissions 777 from being deleted? We can
do that, too. We’ll soon learn that it’s the directory that determines whether a file can
be deleted, not the file itself. Table 4.2 shows the use of chmod both with and without
using octal notation.

Note

4.3.3

A file’s permissions can only be changed by the owner (understood by chmod as user) of the
file. One user can’t change the protection modes of files belonging to another user. However,
this restriction doesn’t apply to the privileged user, root.

Recursive Operation (-R)
chmod -R descends a directory hierarchy and applies the expression to every file and
subdirectory it finds in the tree-walk:
chmod -R a+x shell_scripts

So, to use chmod on your home directory tree, “cd” to it and use it in one of these ways:
chmod -R 755 .
chmod -R a+x *

TA B L E 4 . 2

das76205_Ch04_092-121.indd 99

Works on hidden files also
Leaves out hidden files

chmod Usage

Initial Permissions

Symbolic Expression

Octal Expression

Final Permissions

rw-r----rw-r--r-rwx-----rwxrw--wx
--------r--r--r-rw-rw-rwrw-rw-rw--------rwxrwxrwx

o+rw
u-w,go-r
go+rwx
u-rwx,g-rw,o-wx
+r
+w
-w
a-w
u+w,g+rx,o+x
a=r

646
400
777
000
444
644
466
444
251
444

rw-r--rwr-------rwxrwxrwx
--------r--r--r-rw-r--r-r--rw-rwr--r--r--w-r-x--x
r--r--r--

(Note this)
(Note this)

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

100

When you know the shell metacharacters well, you’ll appreciate the difference between
the two invocations. The dot is generally a safer bet, but note that both commands change
the permissions of directories also. What do permissions mean when they are applied
to a directory? Just read on.

4.4 The Directory
A directory stores the filename and inode number. So the size of a directory is determined
by the number of files housed by it and not by the size of the files. A directory also has
its own set of permissions whose significance differs a great deal from ordinary files.
Let’s use the same umask setting and then create a directory:
$ umask 022 ; mkdir progs ; ls -ld progs
drwxr-xr-x 2 romeo
metal
512 Aug 16 09:24 progs

All categories have read and execute permissions and only the user has write permission.
A directory’s permissions also affect the access rights of its files. This area is the source
of a great deal of confusion, so let’s examine these permissions carefully.
4.4.1

Read Permission
Read permission for a directory means that the list of filenames stored in that directory
is accessible. Since ls reads the directory to display filenames, if a directory’s read
permission is removed, ls won’t work. Consider removing the read permission first
from the directory progs:
$ chmod u-r progs
$ ls progs
progs: Permission denied

However, this doesn’t prevent you from reading the files separately if you know their
names.
4.4.2

Write Permission
Write permission for a directory implies that you are permitted to create or remove
files in it (that would make the kernel modify the directory entries). Security issues are
usually related to a directory’s write permission, so let’s try out a couple of tests with
the directory that we just created.
First, we’ll restore the read permission and then copy a file with permissions 644
to this directory:
chmod u+r progs ; cp date.sh progs

Now let’s “cd” to this directory and display the listing both of the directory and the
filename in it:
$ cd progs ; ls -ld . date.sh
drwxr-xr-x 2 romeo
metal
-rw-r--r-- 1 romeo
metal

das76205_Ch04_092-121.indd 100

512 Aug 16 09:39 .
5 Aug 16 09:39 date.sh

12/13/11 10:44 AM

Chapter 4: File Attributes

101

Both file and directory have write permission for the user. date.sh can now be both
edited and deleted. We can also create a new file in this directory.
Directory’s Write Permission Off; File’s Write Permission On Let’s remove the
directory’s write permission and then check whether we can delete the file:
$ chmod u-w . ; ls -ld . ; rm date.sh
dr-xr-xr-x 2 romeo
metal
512 Aug 16 09:59 .
rm: date.sh not removed: Permission denied

Removing a file implies deletion of its entry from the directory. It’s obvious that date.sh
can’t be deleted. But can it be edited? Yes, of course; the file has write permission, which
means you can edit it with your vi editor. Modifying a file doesn’t affect its directory
entry in any way.
Directory’s Write Permission On; File’s Write Permission Off We now reverse the
previous setting by restoring the directory’s write permission and removing it from the file:
$ chmod u+w . ; chmod u-w date.sh ; ls -ld . date.sh
drwxr-xr-x 2 romeo
metal
512 Aug 16 09:59 .
-r--r--r-- 1 romeo
metal
5 Aug 16 09:39 date.sh

We can create a file in this directory, that’s obvious, but can we delete date.sh?
$ rm date.sh
rm: date.sh: override protection 444 (yes/no)? yes

rm turns interactive when it encounters a file without write permission. Note that the
absence of write permission in date.sh only implies that it can’t be modified. But
whether it can be deleted or not depends entirely on the directory’s permissions.
Directory’s Write Permission Off; File’s Write Permission Off Now that date.sh
is gone, let’s get it again from the parent directory and then switch off the write permission for both file and directory:
$ cp ../date.sh .
$ chmod u-w date.sh . ; ls -ld . date.sh
dr-xr-xr-x 2 romeo
metal
512 Aug 16 10:11 .
-r--r--r-- 1 romeo
metal
5 Aug 16 10:11 date.sh

This is the safest arrangement you can have. You can neither edit the file nor create or
remove files in this directory.
We can now summarize our observations in this manner:
• The write permission for a directory determines whether you can create or remove
files in it because these actions modify the directory.
• Whether you can modify a file depends solely on whether the file itself has write
permission. Changing a file doesn’t modify its directory entry in any way.

das76205_Ch04_092-121.indd 101

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

102

The term “write-protected” has a limited meaning in the UNIX file system. A write-protected
file can’t be written, but it can be removed if the directory has write permission.
Note

Caution

4.4.3

Danger arises when you mistakenly assign the permissions 775 or 777 to a directory. In the
present scenario, 775 allows any user of the metal group to create or remove files in the
directory. 777 extends this facility to the world. As a rule, you must never make directories
group- or world-writable unless you have definite reasons to do so. Sometimes, you’ll have a
good reason (19.4.3).

Execute Permission
Executing a directory just doesn’t make any sense, so what does its execute privilege
mean? It only means that a user can “pass through” the directory in searching for subdirectories. When you use a pathname with any command:
cat /home/romeo/progs/date.sh

you need to have execute permission for each of the directories in the pathname. The
directory home contains the entry for romeo, and the directory romeo contains the entry
for progs, and so forth. If a single directory in this pathname doesn’t have execute
permission, then it can’t be searched for the name of the next directory. That’s why the
execute privilege of a directory is often referred to as the search permission.
A directory has to be searched for the next directory, so the cd command won’t
work if the search permission for the directory is turned off:
$ chmod 666 progs ; ls -ld progs
drw-rw-rw- 2 romeo
metal
512 Aug 16 10:11 progs
$ cd progs
bash: cd: progs: Permission denied

As for regular files, directory permissions are extremely important because system
security is heavily dependent upon them. If you tamper with the permissions of your
directories, then make sure you set them correctly. If you don’t, then be assured that an
intelligent user could make life miserable for you!

4.5 umask: Default File and Directory Permissions
When you create files and directories, the permissions assigned to them depend on the
system’s default setting. The UNIX system has the following default permissions for
all files and directories:
• rw-rw-rw- (octal 666) for regular files.
• rwxrwxrwx (octal 777) for directories.
However, you don’t see these permissions when you create a file or a directory. Actually,
this default is transformed by subtracting the user mask from it to remove one or more

das76205_Ch04_092-121.indd 102

12/13/11 10:44 AM

103

Chapter 4: File Attributes
TA B L E 4 . 3

Effect of umask Settings on Default Permissions

umask Value

Default File Permissions

Default Directory Permissions

000
002
022
026
046
062
066
222
600
666
777

rw-rw-rwrw-rw-r-rw-r--r-rw-r----rw--w---rw----r-rw------r--r--r----rw-rw-----------------

rwxrwxrwx
rwxrwxr-x
rwxr-xr-x
rwxr-x--x
rwx-wx--x
rwx--xr-x
rwx--x--x
r-xr-xr-x
--xrwxrwx
--x--x--x
---------

permissions. To understand what this means, let’s evaluate the current value of the mask
by using umask without arguments:
$ umask
022

This is an octal number which has to be subtracted from the system default to obtain
the actual default. This becomes 644 (666 – 022) for ordinary files and 755 (777 – 022)
for directories. When you create a file on this system, it will have the permissions
rw-r--r--. A directory will have the permissions rwxr-xr-x.
umask is a shell built-in command. A user can also use this command to set a new
default. Here’s an extreme setting:
umask 000

All read-write permissions on

A umask value of 000 means that you haven’t subtracted anything, and this could be
dangerous. The system’s default then applies (666 for files and 777 for directories). All
files and directories are then writable by all; nothing could be worse than that! However,
a mask value of 666 or 777 doesn’t make much sense either; you’ll then be creating files
and directories with no permissions.
The important thing to remember is that no one—not even the administrator—
can use umask to turn on permissions not specified in the systemwide default settings.
However, you can always use chmod as and when required. The systemwide umask
setting is placed in one of the machine’s startup scripts and is automatically made available to all users. The effect of some of these settings on file and directory permissions
is shown in Table 4.3.

4.6

File Systems and Inodes
Before we take up links, we need some idea of the way files are organized in a UNIX
system. So far, we have been referring to the UNIX file hierarchy as a “file system” as
if all files and directories are held together in one big superstructure. That is seldom

das76205_Ch04_092-121.indd 103

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

104

the case, and never so in large systems. The hard disk is split up into distinct partitions
(or slices), with a separate file system in each partition (or slice).
Every file system has a directory structure headed by root. If you have three file systems, then you are dealing with three separate root directories. One of these file systems
is called the root file system, which is more equal than others in at least one respect:
its root directory is also the root directory of the combined UNIX system. The root file
system contains most of the essential files of the UNIX system. At the time of booting,
the other file systems mount (attach) themselves to the root file system, creating the
illusion of a single file system to the user.
Every file is associated with a table called the inode (shortened from index node).
The inode is accessed by the inode number and contains the following attributes of a file:
•
•
•
•
•
•
•
•
•
•

File type (regular, directory, device, etc.)
File permissions (the nine permissions and three more)
Number of links (the number of aliases the file has)
The UID of the owner
The GID of the group owner
File size in bytes
Date and time of last modification
Date and time of last access
Date and time of last change of the inode
An array of pointers that keep track of all disk blocks used by the file

Observe that the inode doesn’t store either the name of the file or the inode number. Both
attributes are stored in the directory. ls displays the inode number with the -i option:
$ ls -i date.sh
254414 date.sh

Every file system has its own set of inodes stored in a separate area of the disk. Since
a UNIX machine usually comprises multiple file systems, you can conclude that the
inode number for a file is unique in a single file system.

How cat and ls Work
When you run cat foo, the kernel first locates the inode number of foo from
the current directory. Next, it reads the inode for foo to fetch the file size and the
addresses of the disk blocks that contain the file’s data. It then goes to each block
and reads the data until the number of characters displayed is equal to the file size.
When you execute ls -l progs where progs is a directory, the kernel looks
up the directory progs and reads all entries. For every entry, the kernel looks up
the inode to fetch the file’s attributes.

das76205_Ch04_092-121.indd 104

12/13/11 10:44 AM

105

Chapter 4: File Attributes

Three More Permission Bits
So far, we have restricted our discussions to nine permission bits. But the inode
stores 12 permission bits. We’ll be discussing the remaining three bits at different
points in the text, but a brief discussion is presented here for completeness.
After completing Chapter 7, you’ll appreciate that file permissions actually
apply to the process run by the user. When you run cat foo, a process named cat
is created from the cat program. Even though the cat executable is owned by root,
the UID of the cat process is that of the user running the program.
This scheme works in most instances except for certain critical events. A
process must sometimes take on the powers of the owner of the program, especially
in situations where the owner is root. The passwd command (which changes your
own password) modifies the file /etc/shadow even though the file is unreadable
to nonprivileged users. Thus passwd must run as if it’s executed by the superuser.
Two permission bits determine whether a process will run with the UID
and GID of the owner and group owner. These two bits, set-user-id (SUID) and
set-group-id (SGID), can be set with chmod, and we’ll reexamine them in Chapter 19.
The third bit, the sticky bit, also applies to a file, but today it is more useful
when set on a directory. Such directories can be shared by a group of users in a
safe manner so that they can create, modify, and remove their own files and not
those of others. The sticky bit is also taken up in Chapter 19.

4.7 ln: Creating Hard Links
Why is the filename not stored in the inode? So that a file can have multiple filenames.
When that happens, we say the file has more than one link. We can then access the file
by any of its links. A file’s link count is normally one, but observe from Fig. 4.1 that
backup.sh has two links:
-rwxr-xr--

2 romeo

metal

163 Jul 13 21:36 backup.sh

The ln command links a file, thus providing it with an alias and increasing the link
count by one. This count is maintained in the inode. ln can create both a hard and a
soft link (discussed later) and has a syntax similar to the one used by cp. Before we use
it to create a hard link, let’s recall the listing of date.sh which we used in Section 4.2,
and then link it with who.sh:
$ ls -li date.sh
254414 -rw-r--r-- 1 romeo
$ ln date.sh who.sh
$ ls -il date.sh who.sh
254414 -rw-r--r-- 2 romeo
254414 -rw-r--r-- 2 romeo

das76205_Ch04_092-121.indd 105

-i displays inode number

metal

5 Aug 16 09:38 date.sh
Link count increases to 2

metal
metal

5 Aug 16 09:38 date.sh
5 Aug 16 09:38 who.sh

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

106

Prior to the invocation of ln, the current directory had an entry containing date.sh and
its inode number, 254414. After the invocation, the kernel performed two tasks:
• It added an entry to the directory for the filename who.sh, but with the same inode
number.
• It also updated the link count in the inode from one to two.
Note that there’s actually one file and that we can’t refer to them as two “files,” but only
as two “filenames.” Changes made in one alias (link) are automatically available in the
others. If you create one more link (using, for example, ln who.sh ps.sh), another
directory entry will be created and the link count will be incremented to three.
ln won’t work if the destination filename exists, but you can force linking with
the -f option. You can also use ln with multiple files (i.e., create a link for each), but
then the destination filename must be a directory. Here’s how you create links for all
shell scripts in the directory shell_scripts:
ln *.sh shell_scripts

If *.sh matches 27 filenames, then there will be 27 linked filenames in shell_scripts;
i.e., there will be 27 entries in that directory.
We use rm to remove files. Technically speaking, rm simply reverses the action of
ln; the kernel removes the directory entry for the link and brings down the link count
in the inode. The following command removes one link:
$ rm who.sh ; ls -l date.sh
254414 -rw-r--r-- 1 romeo

metal

5 Aug 16 09:38 date.sh

The link count has come down to one. Another rm will further bring it down to zero. A file
is considered to be completely removed from the system when its link count drops to zero.
rm and ln are complementary, which is evident from the names of the system
calls they use—unlink and link. The effect they have on the inode and directory is
depicted in Fig. 4.4.
F I G U R E 4.4

Effect of ln and rm on Inode and Directory

Directory

Directory

Directory

Filename

Inode
Number

Filename

Inode
Number

Filename

Inode
Number

.
..
date.sh

386444
417585
254414

.
..
date.sh
who.sh

386444
417585
254414
254414

.
..
date.sh

386444
417585
254414

Inode for date.sh

Inode for date.sh

Inode for date.sh

Link Count = 1

Link Count = 1

Link Count = 2
ln date.sh who.sh

das76205_Ch04_092-121.indd 106

rm who.sh

12/13/11 10:44 AM

107

Chapter 4: File Attributes
4.7.1

Where to Use Hard Links
Links are an interesting feature of the file system, but where does one use them? We
can think of three situations straightaway:
1. Let’s assume that you have written a number of programs that read a file foo.txt
in $HOME/input_files. Later, you reorganize your directory structure and move
foo.txt to $HOME/data instead. What happens to all the programs that look for
foo.txt at its original location? Simple, just link foo.txt to the directory input_files:
ln data/foo.txt input_files

Creates link in directory input_files

With this link available, your existing programs will continue to find foo.txt
in the input_files directory. It’s more convenient to do this than to modify all
programs to point to the new path.
2. Links provide some protection against accidental deletion, especially when they
exist in different directories. Referring to the previous application, even though
there’s only a single file foo.txt on disk, you have effectively made a backup of
this file. If you inadvertently delete input_files/foo.txt, one link will still be
available in data/foo.txt; your file is not gone yet.
3. Because of links, we don’t need to maintain two programs as two separate disk
files if there is very little difference between them. A file’s name is available to a
C program (as argv[0]) and to a shell script (as $0). A single file with two links
can have its program logic make it behave in two different ways depending on the
name by which it is called. There’s a shell script using this feature in Section 13.8.2.
Many UNIX commands are linked. Refer to Section 3.23, where we posed the question whether gzip and gunzip were two separate files. This question can now easily be
answered by looking at their inode numbers:
$ cd /usr/bin ; ls -li gzip gunzip
13975 -r-xr-xr-x 3 root
bin
13975 -r-xr-xr-x 3 root
bin

60916 Jan 5 2000 gunzip
60916 Jan 5 2000 gzip

They are, in fact, one and the same file. The listing shows the existence of a third link as
well, but how does one locate it? Doing an ls -li and then looking for entries with the
same inode number may not always work; a link could be available in another directory.
The find command that can do this job is discussed in Section 4.11.

4.8 ln Again: Creating Symbolic Links
To understand why we need symbolic links, let’s extend the example that we considered as
the first point in Section 4.7.1. Imagine that a hundred files in the directory input_files
have been moved to the directory data as part of the reorganization process. To ensure
that all programs still “see” the files at their original location, we could hard-link these
files to the new data directory, but that would mean adding a hundred entries to this
directory. It’s here that one encounters two serious limitations of hard links:
• You can’t link a file across two file systems. In other words, if input_files and
data are on two separate file systems, you can’t connect a file in one file system

das76205_Ch04_092-121.indd 107

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

108

with a link to the other. This is quite obvious because a hard link takes only an inode
number. There’s no way of knowing which file system the inode number refers to.
• You can’t link a directory even within the same file system.
A symbolic link overcomes both problems. Until now, we have divided files into three
categories (ordinary, directory, and device); the symbolic link is the fourth file type.
Observe the listing in Fig. 4.1 yet again to locate the file hex.c:
lrwxrwxrwx

1 romeo

metal

17 Aug 11 00:49 hex.c -> c_progs/hexdump.c

A symbolic link is identified by the l (el) as the file type and the pointer notation, ->,
that follows the filename. The ln command creates symbolic links also, but it needs
the -s option. We can create a symbolic link to date.sh, but this time the listing tells
you a different story:
$ ln -s date.sh date.sym
$ ls -li date.sh date.sym
254414 -rw-r--r-- 1 romeo
254411 lrwxrwxrwx 1 romeo

metal
metal

5 Aug 16 09:38 date.sh
7 Aug 18 06:52 date.sym -> date.sh

Here, date.sym is a symbolic link to date.sh. Unlike a hard link, a symbolic link is a
separate file with its own inode number. date.sym simply contains the pathname date.sh
as is evident from the file size (date.sh contains seven characters). The two files are not
identical; it’s date.sh that actually has the contents. A command like cat date.sym
follows the symbolic link and displays the file the link points to.
A symbolic link can also point to an absolute pathname, but to ensure portability,
we often make it point to a relative pathname:
$ ln -s ../jscript/search.htm search.htm
$ ls -l search.htm
lrwxrwxrwx 1 romeo metal 21
Mar 2 00:17 search.htm -> ../jscript/search.htm

To return to the problem of linking a hundred files in the directory data, you can use
ln to connect data to a symbolic link named input_files:
ln -s data input_files

First argument is a directory

Being more flexible, a symbolic link is also known as a soft link or symlink. As for
a hard link, the rm command removes a symbolic link even if it points to a directory.
Symbolic links are used extensively in the UNIX system. System files constantly
change locations with version enhancements. Yet it must be ensured that all programs
still find the files where they originally were.
Windows shortcuts are more like symbolic links. The concept of symbolic links can
also be seen in iPod and MP3 players. These devices let you create playlists where the
same song may be present in more than one playlist. Playlist entries resemble symbolic
links in that when you click on an entry, the link to the original file is followed. Also,
deleting a playlist entry doesn’t delete the original MP3 file.

das76205_Ch04_092-121.indd 108

12/13/11 10:44 AM

109

Chapter 4: File Attributes

What happens when we copy a symbolic link with the cp command? By default,
the file pointed to by the symlink is copied and not the symlink itself. But this behavior
can be overridden with the -P option to cp. For instance, if foo.sym is a symlink to the
file foo, then cp -P foo.sym foo.sym2 copies the symlink itself. In this case, both
foo.sym and foo.sym2 point to the same file foo.
A symbolic link has an inode number separate from the file that it points to. In most
cases, the pathname is stored in the symbolic link and occupies space on disk. However,
Linux uses a fast symbolic link, which stores the pathname in the inode itself provided
it doesn’t exceed 60 characters.

Linux

Caution

Think twice before you delete the file or directory that a symlink points to. For instance,
removing date.sym (considered in a previous example) won’t affect us much because we can
easily re-create the link. But if we remove date.sh, we will lose the file containing the data:

$ rm date.sh ; cat date.sym
cat: cannot open date.sym
A disaster of greater magnitude could occur if we removed data instead of input_files. We
would then lose all one hundred files! In either case, date.sym and input_files would point
to a nonexistent file. These links are known as dangling symbolic links.
The pwd command is built into most shells (except the C shell). When you use cd with a symbolic link in one of these shells, pwd shows you the path you used to get to the directory. This
is not necessarily the same as the actual directory you are in. To know the “real” location, you
should use the external command /bin/pwd.

Note

4.9 File Ownership
The chmod and ln commands will fail if you don’t have the authority to use them, that
is, if you don’t own the file. Observe the listing of this file:
rw-r--r--

1 romeo

metal

5 Aug 16 09:38 date.sh

romeo can change all attributes of date.sh, but juliet cannot—even if she belongs to
the metal group. But if juliet copies this file to her home directory, then she’ll be the
owner of the copy and can then change all attributes of the copy at will.
Several users may belong to a single group. People working on a project are
generally assigned a common group, and all files created by group members (who have
separate user-ids) have the same group owner. However, make no mistake: The privileges
of the group are set by the owner of the file and not by the group members.

das76205_Ch04_092-121.indd 109

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

110
F I G U R E 4.5

The UID and GID Components in /etc/passwd

romeo:x:1003:101::/export/home/romeo:/usr/bin/ksh
UID

GID

When the system administrator creates a user account, she has to assign these
parameters to the user:
• The user-id (UID)—both its name and numeric representation.
• The group-id (GID)—both its name and numeric representation. The administrator
has to assign the group name also if the GID represents a new group.
The file /etc/passwd maintains three out of the four parameters. They are shown in
Fig. 4.5 for a sample entry for the user romeo. The UID is shown in the first field (the
name) and the third (the number). The fourth field signifies the GID (number only). The
group database is maintained in /etc/group and contains the GID (both number and
name). The inode, however, stores only the numbers, and commands like ls use these
files as translation tables to display the names rather than the numbers. We’ll discuss
these two files when we add a user account in Chapter 19.
To know your own UID and GID without viewing /etc/passwd and /etc/group, use the id
command:
Tip

$ id
uid=1003(romeo) gid=101(metal)
Whatever files this user creates will have romeo as the owner and metal as the group owner.

4.9.1

chown: Changing File Ownership
There are two commands meant to change the ownership of a file or directory—chown and
chgrp. UNIX systems differ in the way they restrict the usage of these two commands.
On BSD-based systems, only the system administrator can change a file’s owner with
chown. On the same systems, the restrictions are less severe when it comes to changing
groups with chgrp. On other systems, only the owner can change both.
We’ll first consider the behavior of BSD-based chown (change owner) that has been
adopted by many systems, including Solaris and Linux. The command is used in this way:
chown options owner [:group] file(s)

chown transfers ownership of a file to a user, and the syntax shows that it can change
the group as well. The command requires the (UID) of the recipient, followed by one or

das76205_Ch04_092-121.indd 110

12/13/11 10:44 AM

111

Chapter 4: File Attributes

more filenames. Changing ownership requires superuser permission, so let’s first change
our status to that of superuser with the su command:
$ su
Password: ********
# _

This is the root password!
This is another shell

su lets us acquire superuser status if we know the root password, and it returns a # prompt,
the same prompt used by root. To now renounce the ownership of the file date.sh to
juliet, use chown in the following way:
# ls -l date.sh
-rw-r--r-- 1 romeo
metal
5 Aug 18 09:23 date.sh
# chown juliet date.sh ; ls -l date.sh
-rw-r--r-- 1 juliet metal
5 Aug 18 09:23 date.sh
# exit
Switches from superuser’s shell
$ _
to user’s login shell

Once ownership of the file has been given away to juliet, the user file permissions that
previously applied to romeo now apply to juliet. Thus, romeo can no longer edit date.sh
since there’s no write privilege for group and others. He can’t get back the ownership,
either. But he can copy this file, in which case he becomes the owner of the copy.
4.9.2

chgrp: Changing Group Owner
By default, the group owner of a file is the group to which the owner belongs. The chgrp
(change group) command changes a file’s group owner. On systems that implement the
BSD version of chgrp (like Solaris and Linux), a user can change the group owner of a
file, but only to a group to which she also belongs. Yes, a user can belong to more than
one group, and the one shown in /etc/passwd is the user’s main group. We’ll discuss
supplementary groups in Chapter 19 featuring system administration.
chgrp shares a similar syntax with chown. In the following example, romeo changes
the group ownership of a file to dba (no superuser permission required):
$ ls -l prime.c
-r--r--r-- 1 romeo
metal
268 Jul 13 21:36 prime.c
$ chgrp dba prime.c ; ls -l prime.c
-r--r--r-- 1 romeo
dba
268 Jul 13 21:36 prime.c

This command will work on a BSD-based system if romeo is also a member of the dba
group. If he is not, then only the superuser can make the command work. Note that romeo
can reverse this action and restore the previous group ownership (to metal) because he
is still owner of the file and consequently retains all rights related to it.
Using chown to Do Both As an added benefit, UNIX allows the administrator to use
only chown to change both owner and group. The syntax requires the two arguments to
be separated by a :
chown juliet:dba prime.c

das76205_Ch04_092-121.indd 111

Ownership to juliet, group to dba

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

112

Like chmod, both chown and chgrp use the -R option to perform their operations in a
recursive manner.

Tip

4.9.3

If you want members of a project to be able to read and write a set of files, ask the system
administrator to have a common group for them and then set the permissions of the group to
rwx. There’s a better way of doing this (with the sticky bit), and it is discussed in Section 19.4.3.

How to Handle Intruders
View the original listing in Fig. 4.1 to detect two intruders in romeo’s directory. Neither
file is owned by romeo or group-owned by metal:
-r--r--r--rwxr-xr-x

1 256
1 root

105
root

13921 Jul 26 2001 987
6496 Aug 10 10:20 a.out

To explain why numbers rather than names appear in the first line, recall that ls -l does
a number-name translation. It prints the owner’s name by looking up /etc/passwd and
the group name by looking up /etc/group. These numbers are obviously not there in
these files, so ls printed them as they are. Problems of this sort are often encountered
when files are transferred from another system.
For the second file, romeo could have acquired root status to change the ownership
of a file and then forgotten to revert to the normal user before compiling a C program.
To remedy this situation, romeo needs to use the superuser account to run chown and
chgrp on this file or ask the system administrator to do that job for him.

4.10

Modification and Access Times
The inode stores three time stamps. In this section, we’ll be discussing just two of them
(the first two of the following list):
• Time of last file modification
• Time of last access
• Time of last inode modification

Shown by ls -l
Shown by ls -lu
Shown by ls -lc

Whenever you write to a file, the time of last modification is updated in the inode. A
file also has an access time, i.e., the last time someone read, wrote, or executed the file.
This time is distinctly different from the modification time that gets set only when the
contents of the file are changed. ls -l displays the last modification time, and ls -lu
displays the last access time.
A directory can be modified by changing its entries—by creating, removing,
and renaming files in the directory. Note that changing a file’s contents only changes
its last modification time but not that of its directory. For a directory, the access time
is changed by a read operation only; creating or removing a file or doing a “cd” to a
directory doesn’t change its access time.
Even though ls -l and ls -lu show the time of last modification and access,
respectively, the sort order remains standard, i.e., ASCII. However, when you add the -t
option to -l or -lu, the files are actually displayed in order of the respective time stamps:
ls -lt
ls -lut

das76205_Ch04_092-121.indd 112

Displays listing in order of their modification time
Displays listing in order of their access time

12/13/11 10:44 AM

113

Chapter 4: File Attributes

Many tools used by the system administrator look at these time stamps to decide whether
a particular file will participate in a backup or not. A file is often incorrectly stamped
when it is extracted from a backup with tar or cpio. Section 19.3.1 discusses how the
touch command is used to rectify such situations.

Note

Tip

It’s possible to change the access time of a file without changing its modification time. In an
inverse manner, when you modify a file, you generally change its access time as well. However,
on some systems, when you redirect output (with the > and >> symbols), you change the
contents but not the last access time.
What happens when you copy a file with cp? By default, the copy has the modification and
access time stamps set to the time of copying. Sometimes, you may not like this to happen.
In that case, use cp -p (preserve) to retain both time stamps.

4.11 find: Locating Files
find is one of the power tools of the UNIX system. It recursively examines a directory
tree to look for files matching some criteria and then takes some action on the selected
files. It has a difficult command line, and if you have ever wondered why UNIX is hated
by many, then you should look up the cryptic find documentation. However, find is
easily tamed if you break up its arguments into three components:
find path_list selection_criteria action

Fig. 4.6 shows the structure of a typical find command. The command completely
examines a directory tree in this way:
• First, it recursively examines all files in the directories specified in path_list. Here,
it begins the search from /home.
• It then matches each file for one or more selection_criteria. This always consists of
an expression in the form -operator argument (-name index.html). Here, find
selects the file if it has the name index.html.
• Finally, it takes some action on those selected files. The action -print simply
displays the find output on the terminal.
All find operators (also referred to as options in this text) begin with a hyphen. You
can provide one or more subdirectories as the path_list and multiple selection_criteria
to match one or more files. This makes the command difficult to use initially, but it is
a program that every user must master since it lets her select files under practically any
condition.
F I G U R E 4.6

Structure of a find command

find

das76205_Ch04_092-121.indd 113

/home

-name index.html

-print

Path list

Selection criteria

Action

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

114

As our first example, let’s use find to locate all files named a.out (the executable
file generated by the C compiler):
$ find / -name a.out -print
/home/romeo/scripts/a.out
/home/andrew/scripts/reports/a.out
/home/juliet/a.out

Since the search starts from the root directory, find displays absolute pathnames. You
can also use relative names in the path list, and find will then output a list of relative
pathnames. Moreover, when find is used to match a group of filenames with a wild-card
pattern, the pattern should be quoted to prevent the shell from looking at it:
find . -name “*.c” -print
find . -name ‘[A-Z]*’ -print

All files with extension .c
Single quotes will also do

The first command looks for all C program source files in the current directory tree.
The second one searches for all files whose names begin with an uppercase letter. You
must not forget to use the -print option because without it, find on UNIX systems
will look for files all right but won’t print the list.

Linux

4.11.1

find in UNIX displays the file list only if the -print operator is used. However, Linux
doesn’t need this option; it prints by default. Linux also doesn’t need the path list; it
uses the current directory by default. Linux even prints the entire file list when used
without any options whatsoever! This behavior is not required by POSIX.

Selection Criteria
The -name operator is not the only operator used in framing the selection criteria; there
are many others (Table 4.4). We’ll consider the selection criteria first, and then the possible actions we can take on the selected files.
Locating a File by Inode Number (-inum) Refer to Section 4.7.1, where we found
that gzip has three links and gunzip was one of them. find allows us to locate files
by their inode number. Use the -inum option to find all filenames that have the same
inode number:
$ find / -inum 13975 -print
Inode number obtained from Section 4.7.1
find: cannot read dir /usr/lost+found: Permission denied
/usr/bin/gzip
/usr/bin/gunzip
/usr/bin/gzcat
“Cats” a compressed file

Now we know what the three links are. Note that find throws an error message when
it can’t change to a directory. Read the following Tip.

das76205_Ch04_092-121.indd 114

12/13/11 10:44 AM

115

Chapter 4: File Attributes
T A B L E 4 . 4 Major Expressions Used by find (Meaning gets reversed when - is
replaced by +, and vice versa)

Tip

Selection Criteria

Selects File

-inum n
-type x
-perm nnn
-links n
-user usname
-group gname
-size +x[c]
-mtime -x
-newer flname
-mmin -x
-atime +x
-amin +x
-name flname
-iname flname
-follow
-prune
-mount

Having inode number n
If of type x; x can be f (ordinary file), d (directory), or l (symbolic link)
If octal permissions match nnn completely
If having n links
If owned by usname
If owned by group gname
If size greater than x blocks (characters if c is also specified) (Chapter 19)
If modified in less than x days
If modified after flname (Chapter 19)
If modified in less than x minutes (Linux only)
If accessed in more than x days
If accessed in more than x minutes (Linux only)
flname
As above, but match is case-insensitive (Linux only)
After following a symbolic link
But don’t descend directory if matched
But don’t look in other file systems

Action

Significance

-print
-ls
-exec cmd
-ok cmd

Prints selected file on standard output
Executes ls -lids command on selected files
Executes UNIX command cmd followed by {} \;
Like -exec, except that command is executed after user confirmation

If you use find from a nonprivileged account to start its search from root, the command will
generate a lot of error messages on being unable to “cd” to a directory. Since you might miss
the selected file in an error-dominated list, the error messages should be directed by using the
command in this way: find / -name typescript -print 2>/dev/null. Note that you can’t do
this in the C shell. Section 6.7 explains the significance of 2>/dev/null.

File Type and Permissions (-type and -perm) The -type option followed by the
letter f, d, or l selects files of the ordinary, directory, and symbolic link type. Here’s
how you locate all directories of your home directory tree:
$ cd ; find . -type d -print 2>/dev/null
.
./.netscape
./java_progs
./c_progs
./c_progs/include
./.ssh

das76205_Ch04_092-121.indd 115

Shows the . also
Displays hidden directories also

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

116

Note that the relative pathname find displays, but that’s because the pathname itself was
relative (.). find also doesn’t necessarily display an ASCII sorted list. The sequence in
which files are displayed depends on the internal organization of the file system.
The -perm option specifies the permissions to match. For instance, -perm 666
selects files having read and write permission for all user categories. Such files are
security hazards. You’ll often want to use two options in combination to restrict the
search to only directories:
find $HOME -perm 777 -type d -print

find uses an AND condition (an implied -a operator between -perm and -type) to
select directories that provide all access rights to everyone. It selects files only if both
selection criteria (-perm and -type) are fulfilled.
Finding Unused Files (-mtime and -atime) Files tend to build up incessantly on
disk. Some of them remain unaccessed or unmodified for months—even years. find’s
options can easily match a file’s modification (-mtime) and access (-atime) times to
select them. The -mtime option helps in backup operations by providing a list of those
files that have been modified, say, in less than two days:
find . -mtime -2 -print

Here, -2 means less than two days. To select from the /home directory all files that have
not been accessed for more than a year, a positive value has to be used with -atime:
find /home -atime +365 -print

Note

4.11.2

+365 means greater than 365 days; -365 means less than 365 days. For specifying exactly
365, use 365.

The find Operators (!, -o, and -a)
There are three operators that are commonly used with find. The ! operator is used
before an option to negate its meaning. So,
find . ! -name “*.c” -print

selects all but the C program files. To look for both shell and perl scripts, use the -o
operator, which represents an OR condition. We need to use an escaped pair of parentheses here:
find /home \( -name “*.sh” -o -name “*.pl” \) -print

The ( and ) are special characters that are interpreted by the shell to run commands in
a group (7.6.2). The same characters are used by find to group expressions using the
-o and -a operators, the reason why they need to be escaped.

das76205_Ch04_092-121.indd 116

12/13/11 10:44 AM

117

Chapter 4: File Attributes

The -a operator represents an AND condition, and is implied by default whenever
two selection criteria are placed together.
4.11.3

Operators of the Action Component
Displaying the Listing (-ls) The -print option belongs to the action component
of the find syntax. In real life, you’ll often want to take some action on the selected
files and not just display the filenames. For instance, you may want to view the listing
with the -ls option:
$ find . -type f -mtime +2 -mtime -5 -ls
-a option implied
475336 1 -rw-r--r-- 1 romeo
users 716 Aug 17 10:31 ./c_progs/fileinout.c

find here runs the ls -lids command to display a special listing of those regular files
that are modified in more than two days and less than five days. In this example, we see
two options in the selection criteria (both -mtime) simulating an AND condition. It’s
the same as using \( -mtime +2 -a -mtime -5 \).
Taking Action on Selected Files (-exec and -ok) The -exec option allows
you to run any UNIX command on the selected files. -exec takes the command to
execute as its own argument, followed by {} and finally the rather cryptic symbols
\; (backslash and semicolon). This is how you can reuse a previous find command
quite meaningfully:
find $HOME -type f -atime +365 -exec rm {} \;

Note the usage

This will use rm to remove all ordinary files unaccessed for more than a year. This can
be a risky thing to do, so you can consider using rm’s -i option. But not all commands
have interactive options, in which case you should use find’s -ok option:
$
<
<
<

find $HOME -type f -atime
mv ... ./archive.tar.gz >
mv ... ./yourunix02.txt >
mv ... ./yourunix04.txt >
.......

+365 -ok mv {} $HOME/safe \;
? y
? n
? y

mv turns interactive with -i but only if the destination file exists. Here, -ok seeks confirmation for every selected file to be moved to the $HOME/safe directory irrespective
of whether the files exist at the destination or not. A y deletes the file.
find is the system administrator’s tool, and in Chapter 19, you’ll see it used for
a number of tasks. It is especially suitable for backing up files and for use in tandem
with the xargs command (see Going Further of Chapter 6).

Note

das76205_Ch04_092-121.indd 117

The pair of {} is a placeholder for a filename. So, -exec cp {} {}.bak provides a .bak extension
to all selected files. Don’t forget to use the \; symbols at the end of every -exec or -ok option.

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

118

SUMMARY
The ls -l command displays the listing containing seven file attributes. ls -ld used
with a directory name lists directory attributes.
A file can have read, write, or execute permission, and there are three sets of
such permissions for the user, group, and others. A file’s owner uses chmod to alter file
permissions. The permissions can be relative or absolute. The octal digit 7 includes read
(4), write (2), and execute permissions (1).
Permissions have different significance for directories. Read permission means
that the filenames stored in the directory are readable. Write permission implies that you
are permitted to create or remove files in the directory. Execute (or search) permission
means that you can change to that directory with the cd command.
The umask setting determines the default permissions that will be used when
creating a file or a directory.
Multiple file systems, each with its own root directory, are mounted at boot time
to appear as a single file system. A file’s attributes are stored in the inode, which is
identified by the inode number. The inode number is unique in a single file system.
A file can have more than one name or link, and is linked with ln. Two linked
filenames have the same inode number. A symbolic link contains the pathname of another
file or directory and is created with ln -s. The file pointed to can reside on another file
system. rm removes both types of links.
Hard links provide protection against accidental deletion, but removing the file
pointed to by a symlink can be dangerous. Both links enable you to write program code
that does different things depending on the name by which the file is invoked.
chown and chgrp are used to transfer ownership and group ownership, respectively.
They can be used by the owner of the file on AT&T systems. On BSD systems, chown
can be used only by the superuser, and a user can use chgrp to change her group to
another to which she also belongs.
A file has three time stamps, including the time of last modification and access.
find looks for files by matching one or more file attributes. A file can be specified
by type (-type), name (-name), permissions (-perm), or by its time stamps (-mtime and
-atime). The -print option is commonly used, but any UNIX command can be run on
the selected files with or without user confirmation (-ls, -exec, and -ok).

SELF-TEST
4.1
4.2
4.3
4.4
4.5

das76205_Ch04_092-121.indd 118

What do you understand by the listing of a file? How will you save the complete
listing of all files and directories (including the hidden ones) in the system?
Show the octal representation of these permissions: (i) r w x r - x r w - ,
(ii) rw-r----- , (iii) --x-w-r--.
What will the permissions string look like for these octal values? (i) 567,
(ii) 623, (iii) 421
What does a group member require to be able to remove a file?
If a file’s permissions are 000, can the superuser still read and write it?

12/13/11 10:44 AM

119

Chapter 4: File Attributes
4.6
4.7
4.8
4.9
4.10
4.11
4.12
4.13
4.14
4.15
4.16
4.17
4.18
4.19
4.20
4.21
4.22

4.23

You removed the write permission of a file from group and others, and yet they
could delete your file. How could that happen?
Try creating a directory in the system directories /bin and /tmp, and explain
your observations.
Copy a file with permissions 444. Copy it again and explain your observations.
How do you ensure that all ordinary files created by you have rw-rw---- as the
default permissions?
How do you display the inode number of a file?
What does the inode store? Which important file attribute is not maintained in
the inode? Where is it stored then?
What do you mean by saying that a file has three links?
How do you remove (i) a hard link, (ii) a symbolic link pointing to a directory?
How do you link all C source files in the current directory and place the links in
another directory, bar?
A symbolic link has the same inode number as the file it is linked to. True or
false?
How do you link foo1 to foo2 using (i) a hard link, (ii) a symbolic link? If you
delete foo1, does it make any difference?
Copy the file /etc/passwd to your current directory and then observe the listing
of the copy. Which attributes have changed?
Where are the UID and GID of a file stored?
How is chown different from chgrp on a BSD-based system when it comes to
renouncing ownership?
Explain with reference to the dot and * what the following commands do:
(i) chown -R project . , (ii) chgrp -R project * .
When you invoke ls -l foo the access time of foo changes. True or false?
View the access time of a file with ls -lu foo before appending the date command output to it using date >> foo. Observe the access time again. What do
you see?
Devise a find command to locate in /docs and /usr/docs all filenames that
(i) begin with z, (ii) have the extension .html or .java.

EXERCISES
4.1
4.2
4.3
4.4

4.5

das76205_Ch04_092-121.indd 119

A file in a file system with a block size of 1024 bytes contains 1026 bytes. How
many bytes of disk space does it occupy?
Does the owner always belong to the same group as the group owner of a file?
Explain the significance of the following commands: (i) ls -ld ., (ii) ls -l .. .
Create a file foo. How do you assign all permissions to the owner and remove
all permissions from others using (i) relative assignment and (ii) absolute assignment? Do you need to make any assumptions about foo’s default permissions?
From the security viewpoint, explain the consequences of creating a file with
permissions (i) 000, (ii) 777.

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

120
4.6

Examine the output of the following two commands on a BSD-based system.
Explain whether romeo can (i) edit, (ii) delete, (iii) change permissions,
(iv) change ownership of foo:
$ who am i ; ls -l foo
romeo
-r--rw---1 sumit

4.7

4.8
4.9

4.10
4.11
4.12

4.13
4.14
4.15
4.16
4.17
4.18
4.19
4.20
4.21

4.22
4.23
4.24

das76205_Ch04_092-121.indd 120

romeo

78 Jan 27 16:57 foo

Assuming that a file’s current permissions are rw-r-xr--, specify the
chmod expression required to change them to (i) rwxrwxrwx, (ii) r--r-----,
(iii) ---r--r-- , (iv) ---------, using both relative and absolute methods of
assigning permissions.
Use chmod -w . and then try to create and remove a file in the current directory.
Can you do that? Is the command the same as chmod a-w foo?
You tried to copy a file foo from another user’s directory, but you got the error
message cannot create file foo. You have write permission in your own
directory. What could be the reason, and how do you copy the file?
What do you do to ensure that no one is able see the names of the files you have?
The command cd bar failed where bar is a directory. How can that happen?
If a file has the permissions 000, you may or may not be able to delete the file.
Explain how both situations can happen. Does the execute permission have any
role to play here?
If the owner doesn’t have write permission on a file but her group has, can she
(i) edit it, (ii) delete it?
If umask shows the value (i) 000, (ii) 002, what implications do they have from
the security viewpoint?
The UNIX file system has many root directories even though it actually shows
one. True or false?
What change takes place in the inode and directory when a filename is connected
by a hard link?
If ls -li shows two filenames with the same inode number, what does that
indicate?
What happens when you invoke the command ln foo bar if (i) bar doesn’t
exist, (ii) bar exists as an ordinary file, (iii) bar exists as a directory?
How can you make out whether two files are copies or links?
Explain two application areas of hard links. What are the two main disadvantages
of the hard link?
You have a number of programs in $HOME/progs which are called by other programs.
You have now decided to move these programs to $HOME/internet/progs. How
can you ensure that users don’t notice this change?
Explain the significance of fast symbolic links and dangling symbolic links.
Explain how ls obtains the (i) filename, (ii) name of owner, (iii) name of group
owner when displaying the listing.
How will you determine whether your system uses the BSD or AT&T version
of chown and chgrp?

12/13/11 10:44 AM

Chapter 4: File Attributes
4.25

4.26
4.27
4.28
4.29

4.30

das76205_Ch04_092-121.indd 121

121
The owner can change all attributes of a file on a BSD-based system. Explain
whether the statement is true or false. Is there any attribute that can be changed
only by the superuser?
What are the three time stamps maintained in the inode, and how do you display
two of them for the file foo?
How can you find out whether a program has been executed today?
Explain the difference between (i) ls -l and ls -lt, (ii) ls -lu and ls -lut.
Use find to locate from your home directory tree all (i) files with the extension
.html or .HTML, (ii) files having the inode number 9076, (iii) directories having
permissions 666, (iv) files modified yesterday. Will any of these commands fail?
Use find to (i) move all files modified within the last 24 hours to the posix
directory under your parent directory, (ii) locate all files named a.out or core
in your home directory tree and remove them interactively, (iii) locate the file
login.sql in the /oracle directory tree, and then copy it to your own directory, (iv) change all directory permissions to 755 and all file permissions to 644
in your home directory tree.

12/13/11 10:44 AM

CHAPTER

5

The vi/vim Editor

N

o matter what work you do with the UNIX system, you’ll eventually write some
C programs or shell (or perl) scripts. You may have to edit some of the system
files at times. For all of this you must learn to use an editor, and UNIX provides a very
old and versatile one—vi. Bill Joy created this editor for the BSD system. The program
is now standard on all UNIX systems. Bram Moolenaar improved it and called it vim (vi
improved). In this text, we discuss vi and also note the features of vim, available in Linux.
Like any editor including pico, vi supports a number of internal commands
for navigation and text editing. It also permits copying and moving text both within a
file and from one file to another. The commands are cryptic but often mnemonic. vi
makes complete use of the keyboard where practically every key has a function. There
are numerous features available in this editor, but a working knowledge of it is all you
are required to have initially. The advanced features of vi are taken up in Appendix D.

Objectives
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Know the three modes in which vi operates for sharing the workload.
Repeat a command multiple times using a repeat factor.
Insert, append, and replace text in the Input Mode.
Save the buffer and quit the editor using the ex Mode.
Perform navigation in a relative and absolute manner in the Command Mode.
The concept of a word as a navigation unit for movement along a line.
Learn simple editing functions like deleting characters and changing the case of text.
Understand the use of operator–command combinations to delete, yank (copy), and
move text.
Copy and move text from one file to another.
Undo the last editing action and repeat the last command.
Search for a pattern, and repeat the search both forward and back.
Replace one string with another.
Master the three-function sequence to (i) search for a pattern, (ii) take some action,
and (iii) repeat the search and action.
Customize vi using the :set command and the file ~/.exrc.
Become familiar with two powerful features available in vim—word completion and
multiple undoing.
Map your keys and define abbreviations (Going Further).

122

das76205_Ch05_122-155.indd 122

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

123

5.1 vi Basics
vi is unlike other editors. It operates in three modes, and specific functions are assigned
to each mode. We can see these modes at work when we add some text to a file. Invoke
vi with the name of a nonexistent file, say sometext:
vi sometext

vi presents you a full screen with the filename shown at the bottom with the qualifier,
[New File]. The cursor is positioned at the top, and all remaining lines of the screen
(except the last) show a ~. You can’t take your cursor there yet; they are nonexistent
lines. The last line is used for running some commands and displaying system-generated
messages.
You are now in the Command Mode. In this mode you pass commands to act on text.
Pressing a key doesn’t show it on screen but could perform a function like moving the cursor
to the next line or deleting a line. You can’t use the Command Mode to enter or replace text.
Now press the key marked i, and you are in the Input Mode, ready to input text.
Subsequent key depressions will now show up on the screen as text input. Insert a few
lines of text, each line followed by [Enter], as shown in Fig. 5.1. If you see something
that shouldn’t be there, backspace to wipe it out. If a word has been misspelled, use
[Ctrl-w] to erase the entire word.
The current line is the one on which the cursor is now located (like the current
directory). At this moment, the last line is the current line. Now press the [Esc] key to
revert to Command Mode. Press it again, and you’ll hear a beep; a beep in vi indicates
that a key has been pressed unnecessarily.
Now that you are in Command Mode, you can move horizontally along the current
line using the keys h and l (el). Press h to take the cursor left and l to take it right. Use
a repeat factor and try 5h and 3l to see whether the cursor moves five spaces left and
three spaces right. The repeat factor is explained in Section 5.2. Simple navigation in
the four directions is discussed in Section 5.6.1.
FIGURE 5.1

Inserting Some Text

You started text insertion by pressing i.[Enter]
Don’t forget to press [Esc] after keying in text.[Enter]
Then use the h and l keys to move the cursor horizontally.[Enter]
Also try using j and k for moving up and down.[Enter]
Note that your text is still in the buffer and not on disk.[Enter]
Finally, use :x[Enter] to save the buffer to disk and quit vi.
~
~
~
~
~
:x[Enter]
Saves work and quits editor

das76205_Ch05_122-155.indd 123

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

124

The entered text hasn’t been saved on disk yet but exists in some temporary storage
called a buffer. To save this buffer, you must switch to the ex Mode or Last Line Mode
(the third mode) by entering a : (colon), which shows up in the last line. Next enter an
x and press [Enter]:
:x[Enter]
“sometext” 6 lines, 232 characters
$ _

Must be in Command Mode first
Quits editor—back to shell prompt

The file is saved on disk, and vi returns the shell prompt. To modify this file, you’ll have to
invoke vi sometext again. But before moving ahead, let’s summarize the modes used by vi:
• Command Mode—The default mode where every key pressed is interpreted as a
command to run on text. Navigation, copying, and deleting text are performed in
this mode. You used h and l in this mode to move the cursor along a line.
• Input Mode— This mode is invoked by pressing one of the keys shown in Table 5.1.
Every key pressed subsequently shows up as text. Pressing [Esc] in this mode takes
vi to Command Mode. We used i as an Input Mode command.
• ex Mode or Last Line Mode—This mode is used for file handling and performing
substitution. vi was originally hard-linked to a line editor named ex; that’s where this
mode got its name from. A : in the Command Mode invokes this mode, and is then
followed by an ex Mode command. vi normally switches to the Command Mode
after the ex Mode command is run, though we used :x to save the buffer and quit vi.
Much of the chapter deals with Command Mode commands where most of the action is.
Some of these commands also have ex Mode equivalents which are sometimes easier to
use. But all three modes also have their own exclusive features, and an editing session
in vi involves constant switching between modes as depicted in Fig. 5.2.
5.1.1

The File .exrc
The default behavior of vi is adequate for novices, but as you get comfortable with it,
you’ll feel the need to customize it. vi reads the file $HOME/.exrc (same as ~/.exrc
TA B L E 5 . 1

das76205_Ch05_122-155.indd 124

Input Mode Commands

Command

Function

i
a
I
A
o
O
rch
R
s
S

Inserts text to left of cursor (Existing text shifted right)
Appends text to right of cursor (Existing text shifted right)
Inserts text at beginning of line (Existing text shifted right)
Appends text at end of line
Opens line below
Opens line above
Replaces single character under cursor with ch (No [Esc] required)
Replaces text from cursor to right (Existing text overwritten)
Replaces single character under cursor with any number of characters
Replaces entire line

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
FIGURE 5.2

125
The Three Modes

ex Mode

Input Mode

[Esc]

[Enter]
:

i,I,a,A,o,O,
r,R,s,S
and c operator

Command
Mode

:x, :wq, :q,
and ZZ
vi foo

Shell

in some shells) on startup. If ls -a doesn’t show this file in your home directory, then
you can create or copy one. Linux users must note that vim generally doesn’t use .exrc,
but only .vimrc.
Many ex Mode commands can be placed in this file so they are available in every
session. You can create abbreviations, redefine your keys to behave differently, and make
variable settings. Your .exrc will progressively develop into an exclusive “library”
containing all shortcuts and settings that you use regularly. It could be your most prized
possession, so always keep a backup of this file.

5.2

A Few Tips First
We are about to take off, but before we do, a few tips at this stage will stand you in
good stead. You must keep them in mind at all times when you are doing work with vi:
• Make use of the repeat factor. Like more (3.16), vi also supports the use of
a repeat factor as a command prefix to repeat the command as many times as
the prefix. So if the Command Mode command k moves the cursor up one line,
then 10k moves it up 10 lines. Use the repeat factor wherever you can to speed
up operations.

das76205_Ch05_122-155.indd 125

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

126

• Undo whenever you make a mistake. If you have made a mistake in editing, either by wrongly deleting text or inserting it at an incorrect location, then as a first
measure, just press [Esc] and then u to undo the last action. If that makes matters
worse, use u again. Linux users should instead use [Ctrl-r].
• Use [Ctrl-l] to clear the screen. If the screen gets garbled, enter this control sequence in the Command Mode to redraw the screen. If you hit [Ctrl-l] in the Input
Mode, you’ll see the symbol ^L on the screen. Use the backspace key to wipe it
out, press [Esc], and then hit [Ctrl-l].
• Don’t use [CapsLock] on most occasions. vi commands are case-sensitive; a and
A are different Input Mode commands. Even if you have to activate [CapsLock]
to enter a block of text in uppercase, make sure you deactivate it after text entry
is complete.
• Don’t use the PC navigation keys Avoid using the keys marked Up, Down, Left,
and Right, [PageUp] and [PageDown]. Many of them could fail when you use
vi over a network connection. vi provides an elaborate set of keys for navigation
purposes.
• vi reads the TERM variable to determine the file that contains the terminal’s characteristics. As discussed later (8.3), vi actually reads a file in a specific directory to
know the control sequences that apply to the terminal name assigned to TERM. You
should always check TERM whenever vi behaves in an awkward manner.
Only the keys g, K, q, v, V, and Z have no function in the standard vi implementation. Some
of them are defined, however, in vim.

Note

5.3

Input Mode—Entering and Replacing Text
In this section, we take up all the commands that let you enter the Input Mode from the
Command Mode. When a key of the Input Mode is pressed, it doesn’t appear on the
screen, but subsequent key depressions do. We’ll consider the following commands:
• Insert and append (i, a, I, and A)
• Replace (r, R, s, and S)
• Open a line (o and O)
Always keep in mind that after you have completed text entry using any of these commands (except r), you must return to the Command Mode by pressing [Esc]. Most of
these commands can also be used with a repeat factor, though you’ll need to use it with
only some of them.
Before you start using the Input Mode commands, enter this ex Mode command:

:set showmode[Enter]

Tip

das76205_Ch05_122-155.indd 126

Enter a : (the ex Mode prompt), and you’ll see it appearing in the last line. Follow it with
the two words and press [Enter]. showmode sets one of the parameters of the vi environment.
Messages like INSERT MODE, REPLACE MODE or CHANGE MODE, etc. will now appear in the last line
when you run an Input Mode command. We’ll learn later to make the setting permanent by
placing it in $HOME/.exrc.

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
5.3.1

127

Inserting and Appending Text (i and a)
The simplest type of input is the insertion of text. Just press
i

Existing text will be shifted right

Pressing this key changes the mode from Command to Input. Since the showmode setting
was made at the beginning (with :set showmode), you’ll see the words INSERT MODE
at the bottom-right corner of the screen. You can now input as much text as you like.
If the i command is invoked with the cursor positioned on existing text, text on
its right will be shifted further without being overwritten. The insertion of text with i is
shown in Fig. 5.3. All figures in this chapter use this shaded box to represent the cursor
and the to signify a space (when its presence isn’t all that obvious).
There are other methods of inputting text. To append text to the right of the
cursor, use
a

Existing text will also be shifted right

followed by the text you wish to key in (Fig. 5.4). After you have finished editing, press
[Esc]. With i and a, you can input several lines of text.
5.3.2

Inserting and Appending Text at Line Extremes (I and A)
I and A behave somewhat like i and a except that they work at line extremes by also
performing the necessary navigation to move there:
I
A

Inserts text at beginning of line.
Appends text at end of line.

FIGURE 5.3
Original Text

vi Commands

Transformed Text

printf(No
N entry);

i”[Esc]

printf(“No
“ entry);

printf(“No entry);
)

i\n”[Esc]

printf(“No entry\n”);
”

if (x < 5) {
break;
b

i[Tab][Esc]

if (x < 5) {
break;

FIGURE 5.4

das76205_Ch05_122-155.indd 127

Inserting Text with i

Appending Text with a

Original Text

vi Commands

Transformed Text

echo “Filename:”
:

a\c[Esc]

echo “Filename:\c”
c

[ -gt 5 ]
if [$x

a [Esc]

if [ $x -gt 5 ]

casee

a $# in[Esc]

case $# inn

“
echo “”

10a*[Esc]

*
echo “**********”

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

128

These two commands are suitable for converting code to comment lines in a C program
(Fig. 5.5). A comment line in C is of the form /* comment */. Use I on an existing line
that you now wish to convert to a comment, and then enter the symbols /*. After pressing
[Esc], use A to append */ at the end of the line and press [Esc] again. A document author
often needs to use A to add a sentence to a paragraph.
C language programmers sometimes precede a block of comment lines with a pattern like this:

/**********************************************************************
Tip

5.3.3

It seems that there are 70 asterisks here, and to draw them you should use a repeat factor.
After you have entered / in the Input Mode, press [Esc], and then enter 70a*[Esc]. You’ll see
70 asterisks appended to the /.

Opening a New Line (o and O)
Often it has been observed that people migrating from a word-processing environment
use a convoluted sequence to open a new line below the current line. They move the
cursor past the end of the current line and then press [Enter]. In vi, you are better off
using o and O instead. To open a line below from anywhere in a line, simply press
o

Opens a new line below the current line

This inserts an empty line below the current line (Fig. 5.6). O also opens a line but above
the current line. Note that to insert a line before the first line, you need O and not o. Press
[Esc] after completing text input.
FIGURE 5.5

Using I and A

Original Text

vi Commands

Transformed Text

Set up an infinite loop

I/* [Esc]

/* Set up an infinite loop

/* Set up an infinite loop A */[Esc]

/* Set up an infinite loop *//

exit(0)

exit(0);

A;[Enter]}[Esc]

}

FIGURE 5.6
Original Text

Opening a New Line with o and O
vi Commands

#count.sh: Counts patterns o#[Esc]

Transformed Text

#count.sh: Counts patterns
#

#count.sh: Counts patterns O#!/bin/sh[Esc]
#

das76205_Ch05_122-155.indd 128

#!/bin/sh
#count.sh: Counts patterns
#

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
5.3.4

129

Replacing Text (r, s, R, and S)
To change existing text, vi provides mainly four commands (actually, more) as shown
in the heading. To replace a single character with another, you should use
r

No [Esc] required

followed by the character that replaces the one under the cursor (Fig. 5.7). You can replace
a single character only in this way. vi momentarily switches from Command Mode to
Input Mode when r is pressed. It returns to the Command Mode as soon as the new
character is entered. There’s no need to press [Esc] when using r and the replacement
character, since vi expects a single character anyway.
When you want to replace the letter d with 10f in a printf statement in C, you
need to replace one character with three. In that case, press
s

Replaces one character with many

vi deletes the character under the cursor and switches to Input Mode (Fig. 5.8). It may
also show a $ at that location to indicate that replacement will not affect text on its right.
Now enter 10f and press [Esc]. To replace multiple characters, use a repeat factor. 3s
replaces three characters with new text.
R and S act in a similar manner compared to their lowercase versions except that
they act on a larger group of characters:
R
S

Replaces all text on the right of the cursor position.
Replaces the entire line irrespective of the cursor position. (Existing line disappears)

Using R in vi is like using [Insert] to activate the overwrite mode in Windows. Try using
the repeat factor with R and S, and see whether you can take advantage of this feature.
Use of R and S is shown in Fig. 5.9.
FIGURE 5.7

Replacing a Single Character with r

Original Text

vi Commands

Transformed Text

printf(“Enter filename/c”);

r\

printf(“Enter filename\c”);

printf(“Filename: %d\n”, fname);

rs

printf(“Filename: %s\n”, fname)

FIGURE 5.8

Replacing Text with s and S

Original Text

das76205_Ch05_122-155.indd 129

vi Commands Transformed Text

printf(“CPU Time: %d\n”,
d
ct); s10f

printf(“CPU Time: %10f\n”,
f
ct);

- 5)
while (x -gt

3s>[Esc]

while (x > 5)

(
while(1)

3s true[Esc]

while truee

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

130
FIGURE 5.9

Replacing Text with R and S

Original Text

vi Commands

Transformed Text

while (1)
(

Rtrue[Esc]

while truee

echo “Error”

Sprintf “Command failed\n”[Esc]

printf “Command failed\n””

You have now been able to enter the Input Mode in 10 ways. The functions of
these 10 keys are summarized in Table 5.1.

Caution

Linux

Remember to switch to the Command Mode from Input Mode by pressing [Esc]. If you forget to
do that, all Command Mode commands will show up as text input, in which case you have to
backspace to erase the text. Repeated pressing of [Esc] won’t make any difference to vi except
that it has a built-in capability to indicate with a beep if a key has been pressed unnecessarily.
Try this by pressing [Esc] several times. You are now in the Command Mode.

A superb text completion feature is available in vim. If the string printf is available
in the file, you don’t need to enter the entire string ever. Just key in as much as is
necessary to make the string unique (say, up to pr), and then press
[Ctrl-p]

vim attempts to complete string

vim expands pr to printf if this is the only word beginning with pr. In case there are
other words, repeated pressing of [Ctrl-f] shows all matching words in turn. In case
you have to view the list backwards, use [Ctrl-n].

5.3.5

Entering Control Characters ([Ctrl-v])
If you write shell scripts to send some escape sequences to your printer or terminal, then
you need to enter control characters. In vi, some of these characters are directly enterable,
but generally a control character has to be preceded by [Ctrl-v] to be interpreted properly.
For instance, to enter [Ctrl-h], you have to first press [Ctrl-v] and then [Ctrl-h].
You’ll then see this on the screen:
^H

Just one character here

Even though you feel you are seeing a ^ (caret) and an H, there’s only a single character
out there. You can position the cursor only on the ^ and not on the H; that’s how you
identify control characters anyway.
The same technique can be adopted for entering the [Esc] character. Press [Ctrl-v][Esc],
and you’ll see the [Esc] character looking like this:
^[

Just one character here

This too is a single character. Insertion of a control character is shown in Fig. 5.10. If
[Ctrl-v][Esc] doesn’t work, then use this: [Ctrl-v][Ctrl-[] (Control with [).

das76205_Ch05_122-155.indd 130

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

131

FIGURE 5.10

5.4

Insertion of a Control Character

Original Text

vi Commands

Transformed Text

:map #1 :w

A[Ctrl-v][Ctrl-m][Esc]

:map #1 :w^M

echo “Time up”

i[Ctrl-v][Ctrl-g]

echo “Time up^G”

Saving Text and Quitting—The ex Mode
When you edit a file using vi—or, for that matter, any editor—the original file isn’t
disturbed as such, but only a copy of it that is placed in a buffer (a temporary form of
storage). These are the three operations that we commonly perform with the buffer:
• Save and continue editing (:w).
• Save and exit (:x and :wq).
• Abandon all changes and quit (:q and :q!).
The associated ex Mode commands are shown in parentheses. From time to time, you
should save your work by writing the buffer contents to disk to keep the disk file current (or, as we say, in sync). When we talk of saving a file, we actually mean saving this
buffer. The essential save and exit commands are shown in Table 5.2.
TA B L E 5 . 2
Command

Action

:w
:x
:wq
:w n2w.pl
:w! n2w.pl
:q
:q!
:n1,n2w build.sql
:3,10w build.sql
:.w build.sql
:$w build.sql
:!cmd
:sh
[Ctrl-z]

Saves file and remains in editing mode
Saves file and quits editing mode
As above
Like Save As ..... in Microsoft Windows
As above, but overwrites existing file
Quits editing mode when no changes are made to file
Quits editing mode after abandoning changes
Writes lines n1 to n2 to file build.sql
Writes lines 3 to 10 to file build.sql
Writes current line to file build.sql
Writes last line to file build.sql
Runs cmd command and returns to Command Mode
Escapes to UNIX shell (use exit to return to vi)
Suspends current session and escapes to UNIX shell (only for shells
supporting job control; use fg to return to vi)
Stops editing current file and edits file note1
Loads last saved edition of current file (Like Revert in
Microsoft Windows)
Returns to editing most recently edited file

:e note1
:e!
:e#

das76205_Ch05_122-155.indd 131

Save and Exit Commands of the ex Mode

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

132
5.4.1

Saving Your Work (:w)
You know how to save the buffer and exit the editor (5.1). For extended sessions with
vi, you must be able to save the buffer and remain in the editor. Enter a : and w:
:w[Enter]
“sometext”, 8 lines, 275 characters

You can now continue your editing work normally; only make sure that you execute
this command regularly. You may want to keep a backup of your work by using :w with
a filename:
:w anotherfile
anotherfile must not exist
“anotherfile” [New File] 8 lines, 275 characters written

Attention! Even though you have just written to anotherfile, your current file continues
to be sometext. Windows users should note that this alternate file saving facility is
different from the Save As . . . option of the File menu, which saves to a different file
but also makes the new file the current one.

Tip

5.4.2

It’s common practice to ignore the readonly label when opening a file that doesn’t have the
write permission bit set. When you attempt to save the file with :w, vi retorts with the message
File is read only. You should have been careful in the first place, but there’s hope: Just save
the file with a different name (say, :w foo) after making sure that foo doesn’t exist. Look up
Table 5.2 for the command to use when foo also exists.

Saving and Quitting (:x and :wq)
You know how to save and quit the editor (i.e., return to the shell); use :x:
:x[Enter]
“sometext”, 8 lines, 303 characters
$ _

You can also use :wq as a synonym to :x. But that requires an additional keystroke, and
this author doesn’t use it.

Tip

5.4.3

The best way to save and quit the editor is to use ZZ, a Command Mode command, instead
of :x or :wq. But there’s a danger that you might hit [Ctrl-z] instead and suspend the process.
If that happens, turn to Section 7.11 for remedial action.

Aborting Editing (:q)
It’s also possible to abort the editing process and quit the editing mode without saving
the buffer. The q (quit) command is used to do that:
:q[Enter]
$ _

das76205_Ch05_122-155.indd 132

Won’t work if buffer is unsaved

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

133

vi also has a safety mechanism that prevents you from aborting accidentally if you
have modified the file (buffer) in any way. The following message is typical when you
try to do so:
No write since last change (:quit! overrides)

You haven’t saved the changes and are trying to quit; that’s what the message says. If
you still want to abandon the changes, then use
:q!

Ignores all changes made and quits

to return you to the prompt irrespective of the status of the buffer—no questions asked.
The ! makes its appearance in the editor’s message every time vi feels that you could
be doing something that is potentially unsafe.

Note

5.4.4

In general, any ex Mode command used with a ! signifies an abort of some type. It can be
used to switch to another file without saving the current one, or reload the last saved version
of a file. You can even use it to overwrite a separate file.

Writing Selected Lines
The :w command is an abbreviated way of executing the ex Mode instruction :1,$w.
The w command can be prefixed by one or two addresses separated by a comma. The
following two ex Mode commands write a single line and 41 lines, respectively, to a
separate file:
:5w n2words.pl
:10,50w n2words.pl

Writes fifth line to another file
Writes 41 lines to another file

There are two symbols used with w that have special significance—the dot and $. The
dot represents the current line while $ represents the last line of the file. You can use
them singly or in combination:
:.w tempfile
:$w tempfile
:.,$w tempfile

Saves current line (where cursor is positioned)
Saves last line
Saves current line through end

If tempfile exists and is writable by you, vi issues yet another warning:
“tempfile” File exists - use “w! tempfile” to overwrite

You know what vi is saying: Use :w! tempfile to overwrite tempfile. The ! is the
universal overriding operator in the ex Mode, and you’ll be using it often.

Note

das76205_Ch05_122-155.indd 133

In the ex Mode, the current line number is represented by . (dot) and the last line is denoted
by $. The command w is the same as 1,$w. Both commands address the entire buffer.

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

134
5.4.5

Escape to the Shell (:sh and [Ctrl-z])
How do you edit and compile your C program repeatedly? You need to make a temporary
escape to the shell to run the cc command. There are two ways; the first method is to
use the ex Mode command, sh:
:sh
$ _

You haven’t quit vi yet

This returns a shell prompt. Execute cc or any UNIX command here and then return to
the editor using [Ctrl-d] or exit. Don’t make the mistake of running vi once again, as
you’ll then have two instances of vi—an undesirable situation. Table 5.2 shows how
you can run a single command using :! and remain in the editor.
The shell that shows its presence by the $ or % prompt is determined by the setting of your
SHELL variable. This is the last field in /etc/passwd and is assigned when the account is opened
(19.3.2). Even though sh actually represents the Bourne shell, :sh is a generic shell escape
command. Just run echo $SHELL to find out the shell you are using.

Note

The second method will work if your shell supports job control (which most shells do).
You can then suspend the current vi session. Just press [Ctrl-z] and you’ll be returned a
shell prompt. Run your commands and then use the fg command to return to the editor.
Job control is discussed in Section 7.11.

5.5

Recovering from a Crash (:recover and -r)
Accidents can and will happen. The power will go off, leaving work unsaved. However,
don’t panic; vi stores most of its buffer information in a hidden swap file. Even though
vi removes this file on successful exit, a power glitch or an improper shutdown procedure lets this swap file remain on disk. vi will then complain the next time you invoke
it with the same filename.
The complaint usually also contains some advice regarding the salvage operation.
You’ll be advised to use either the ex Mode command :recover, or vi -r foo to recover as much of foo as possible. After you have done that, have a look at the buffer’s
contents and satisfy yourself of the success of the damage control exercise. If everything
seems fine, save the buffer and remove the swap file if vi doesn’t do that on its own.

Caution

You can’t be assured of complete recovery every time. You may see junk when using vi -r (or
:recover), in which case, don’t save the file and simply quit (with :q!). Start vi again normally;
recovery is not possible here. Linux users should note that in these situations, they may be
required to delete the file .foo.swp if foo has recovery problems.

5.6 Navigation
We’ll now consider the functions of the Command Mode. This is the mode you come
to when you have finished entering or changing text. We begin with navigation whose

das76205_Ch05_122-155.indd 134

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
TA B L E 5 . 3

135
Navigation and Scrolling Commands

Command

Function

Relative Motion
h (or [Backspace])
l (or spacebar)
5l
k
10k
j

Moves cursor left
Moves cursor right
Moves five characters right
Moves cursor up
Moves 10 lines up
Moves cursor down

Scrolling
[Ctrl-f]
5[Ctrl-f]
[Ctrl-b]
[Ctrl-d]
[Ctrl-u]
[Ctrl-l]

Scrolls full page forward
Scrolls five full pages forward
Scrolls full page back
Scrolls half page forward
Scrolls half page back
Redraws the screen (no repeat factor)

Word Navigation
b
4b
e
w
8w

Moves back to beginning of word
Moves back four words to beginning of word
Moves forward to end of word
Moves forward to beginning of word
Moves forward to beginning of 8th word

Line Navigation and Absolute Movement
0 (zero) or |
30|
^
$
1G
40G
G

Moves to beginning of line
Moves to column 30
Moves to first word in line
Moves to end of line
Moves to beginning of buffer
Moves to line 40
Moves to end of buffer

commands are listed in Table 5.3. Don’t forget to avoid the cursor control keys for
navigation as advised in Section 5.2.
5.6.1

Relative Movement in the Four Directions (h, j, k, and l)
vi provides the h, j, k, and l commands to move the cursor in the four directions. These
keys are placed adjacent to one another in the middle row of the keyboard. Without a
repeat factor, they move the cursor by one position. Use these commands for moving
the cursor vertically:
k
j

das76205_Ch05_122-155.indd 135

Moves cursor up
Moves cursor down

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

136

To move the cursor along a line, you have already used these commands in the preliminary session:
h
l (el)

Moves cursor left
Moves cursor right

The repeat factor can be used as a command prefix with all of these four commands.
Thus, 4k moves the cursor 4 lines up, and 20h takes it 20 characters to the left. Note
that this motion is relative; you can’t move to a specific line number with these keys.
Navigation with the four keys is shown in Fig. 5.11.

Tip

5.6.2

To remember the keys that move the cursor left or right, observe these four keys on your keyboard. The left-most key, h, moves the cursor to the left, and the right-most key, l, moves it right.

Scrolling ([Ctrl-f], [Ctrl-b], [Ctrl-d] and [Ctrl-u])
Faster movement can be achieved by scrolling text in the window using the control keys.
The two commands for scrolling a page at a time are
[Ctrl-f]
[Ctrl-b]

Scrolls forward
Scrolls back

You can use the repeat factor, as in 10[Ctrl-f], to scroll 10 pages and navigate faster in
the process. You can scroll by a half page as well:
[Ctrl-d]
[Ctrl-u]

Scrolls half page forward
Scrolls half page back

The repeat factor can also be used here.
5.6.3

Word Navigation (b, e, and w)
Moving by one character is not always enough, and you’ll often need to move faster
along a line. vi understands a word as a navigation unit which can be defined in two
FIGURE 5.11

Relative Navigation with h, j, k and l
l takes you right and h takes you left
3l

2k

k takes you up
j
j takes you down

13h

das76205_Ch05_122-155.indd 136

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

137

ways, depending on the key pressed. If your cursor is a number of words away from
your desired position, you can use the word-navigation commands to go there directly.
There are three basic commands:
b
w
e

Moves back to beginning of word
Moves forward to beginning of word
Moves forward to end of word

A repeat factor speeds up cursor movement along a line. For example, 5b takes the
cursor five words back, while 3w takes the cursor three words forward. A word here is
simply a string of alphanumeric characters and the _ (underscore). Bash is one word; so
is sh_profile. tcp-ip is three words; the hyphen by definition becomes a word too.
The commands B, E and W perform functions similar to those of their lowercase
counterparts except that they skip punctuation. The word definition also gets changed
here, but we’ll ignore these minor details.
5.6.4

Moving to Line Extremes (0, |, and $)
Moving to the beginning or end of a line is a common requirement. This is handled by
the keys 0, |, and $. To move to the first character of a line, use
0 (zero) or |

30| moves the cursor to column 30

The | takes a repeat factor and by using it, you can position the cursor on a certain
column. To position the cursor on column 30, use 30|.
We used $ as the line address in the ex Mode to represent the last line of the file.
The same symbol in the Command Mode represents the end of line. To move to the end
of the current line, use
$

Moves to end of line

The use of these two commands along with those that use units of words (b, e, and w)
is shown in Fig. 5.12.
5.6.5

Absolute Movement (G)
Upon startup, vi displays the total number of lines in the last line. At any time, you can
press [Ctrl-g] to know the current line number:
“/etc/passwd” [Read only] line 89 of 179 --49%--

The cursor is on line 89 (49% of 179), and this write-protected file has 179 lines in all.
Compilers also indicate line numbers in their error messages. You need to use the G command
with the line number as repeat factor to locate the offending lines. To move to the 40th line, use
40G

Goes to line number 40

and to move to the beginning of the buffer, use
1G

das76205_Ch05_122-155.indd 137

Goes to line number 1

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

138
FIGURE 5.12

Finer Navigation with b, e, w, 0 and $
0 or |
b

You can move to beginning or end of line and also in units of words

e

4e

4w

$

The end of the buffer is reached by simply using
G

Goes to end of file

The ex Mode offers equivalent commands for moving between lines. The previous three commands can be replaced by :40, :1 and :$, respectively (along with [Enter]).

Note

5.7

Editing Text without Operators
Editing operations change the contents of the buffer. These operations are performed
with commands of both the Input and Command Modes. Having seen the Input Mode
commands, let’s turn our attention to the ones used in the Command Mode. The simple
ones don’t use operators, but the powerful ones do. Let’s first use the simple ones before
we take up operator-based editing.

5.7.1

Deleting Text (x, X, and dd)
The x and X commands are used to delete one or more contiguous characters. Move the
cursor to any location and press
x

Deletes a single character

The character under the cursor gets deleted and text on the right shifts left to fill up the
space. A repeat factor also applies here, so 4x deletes the current character as well as
three characters from the right (Fig. 5.13).
A Windows Notepad or Wordpad user would be surprised to note that when the
cursor is at the end of a line, x doesn’t pull up the following line. Instead, it deletes text
on the left of the cursor.
Text deletion to the left is otherwise handled by the X command. Keep it pressed,
and you’ll see that you have erased all text to the beginning of the line.

das76205_Ch05_122-155.indd 138

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
FIGURE 5.13

139
Deleting Text with x

Original Text

vi Commands

Transformed Text

$$x=5

x

x=5
x

#! /usr/bin/sh

x

/
#!/usr/bin/sh

/
#!/usr/bin/sh

4x

/
#!/bin/sh

FIGURE 5.14

Transposing Characters with x and p

Original Text

vi Commands

Transformed Text

sdtio.h
d

x

stio.h
t

t
stio.h

p

d
stdio.h

How about deleting a line? The command to use is a two-character string named
dd. Move the cursor to any point on a line and then use
dd

to delete the current line. Even though we are not discussing operators in this section (dd
is actually a special use of an operator), you need to know this special command because
line deletion is a frequent editing operation. The topic actually belongs to Section 5.9.1.
5.7.2

Moving Text (p)
Text movement requires you to perform an additional task: Put the deleted text at the
new location with p or P. vi uses these two commands for all “put” operations that follow delete or copy operations (even when using operators). The significance of p and P
depends on whether they are used on parts of lines or complete lines.
For instance, to correct sdtio.h to stdio.h, you’ll have to transpose the characters
d and t, i.e., delete the d and put it after the t. Move your cursor to the d in sdtio.h
and then use x and p as shown Fig. 5.14. The sequence xp is easily remembered as a
mnemonic for transpose.
Since we worked on parts of lines, associate the word “right” with p and “left”
with P. P places text on the left of the cursor.

5.7.3

Joining Lines (J)
In word processors, you join the current and next line by moving the cursor to the end
of line and pressing [Delete]. This technique won’t work in vi. To join the current line
and the line following it, use
J

4J joins following 3 lines with current one

J removes the newline character between the two lines to pull up the line below the
current line (Fig. 5.15). Joining, however, is restricted to the maximum line size that your
vi editor can support. It could be around 2000 (as in Solaris) or unlimited (in Linux).

das76205_Ch05_122-155.indd 139

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

140
FIGURE 5.15
Original Text

vi Commands

Transformed Text

int main(void)
{

J

int main(void) {

while true
do

A ;[Esc]J

while true ; do

FIGURE 5.16

5.7.4

Joining Lines with J

Changing Case with ~

Original Text

vi Commands

Transformed Text

if [ “$answer” = “Y” ]

~

if [ “$answer” = “y” ]

mail=/var/mail/romeo

4~

MAIL=/var/mail/romeo

Changing Case (~)
vi uses the ~ (tilde) command to toggle the case of text. To reverse the case of a section
of text, move the cursor to the first character of the section and then press
~

Upper becomes lower, lower becomes upper

If you have to do this for a string of a hundred characters, use a repeat factor: 100~.
The ~ is not suitable for changing an entire line to uppercase if some characters in the
line are already in uppercase; those characters then turn to lowercase. The changing of
case is shown in Fig. 5.16.

5.8

Correcting a C Program
With so many commands at our disposal, let’s go through an editing session that corrects some common mistakes found in C programs. Though the changes required seem
to underestimate the programmer, it’s a great way to learn vi. Fig. 5.17 shows the file
foo.c both before and after correction.
Except for line 2, all of the other lines need to be corrected. Beginners of C programming tend to drop the < and > around the include filename. Single quotes are used
when double quotes are required. It’s common folly both to forget to include the \n in
the printf statement and the ; as the statement terminator. The exit function must be
used with an argument, and the closing brace has to be inserted at the end. Table 5.4
lists the vi sequences that perform this transformation.
Try the commands in the specified sequence after downloading foo.c from the
Web site. If you make a mistake, don’t forget to press u (after [Esc]) to undo the last
editing action. If there are multiple mistakes, simply reload the last saved version with
:e! and then start all over again.

das76205_Ch05_122-155.indd 140

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
FIGURE 5.17

141
foo.c with Errors (Left), after Correction (Right)

#include stdio.h
#include 
int quit(char *message) {
printf(‘Error number %10d’, errno)
exit;

TA B L E 5 . 4

5.9

#include 
#include 
void quit(char* message) {
printf(“Error number %d\n “, errno);
exit(1);
}

vi Commands Required to Correct Program in Fig. 5.17

Command

Action

1G
2w
i<[Esc]
A>[Esc]
2j
3svoid[Esc]
j2e
r”
4w
2x
a\n [Esc]
l
r”
A;[Esc]
j$
i(1)[Esc]
o}[Esc]
:x

Moves to line 1
Moves to s in stdio.h
Inserts <; shows ; shows 
Move two lines below
Replaces int with void
Moves to first ‘ on next line
Replaces ‘ with “
Moves to 1 in 10
Deletes 10 ; shows %d
Appends \n and a space to %d
Moves to closing ‘
Replaces ‘ with “
Appends ; at end of line
Moves to end of next line; at ; in exit;
Inserts (1) after t in exit
Opens line below and inserts }
Saves and quits vi

Editing Text with Operators
Arguably, vi’s strongest feature is its ability to combine an operator with a command to
handle any type of deletion or copying task. This opens up a new domain of commands
for you to work with. In this text, we consider the following operators:
d —Delete
y —Yank (copy)
c —Change
An operator alone can’t perform any function unless it is combined with a command
or itself. When used with itself (i.e., twice), it can handle a very specialized function.

das76205_Ch05_122-155.indd 141

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

142
TA B L E 5 . 5

A Few Specimen Operator-Command Combinations

Command

Function

d$ or D
5dd
d/}
d30G
y$ or Y
3yw or y3w
5yy
yG

Deletes from cursor to end of line
Deletes five lines
Deletes from cursor up to first occurrence of }
Deletes from cursor up to line number 30
Yanks from cursor to end of line
Yanks three words from cursor position
Yanks five lines
Yanks from cursor to end of file

y?case
c0
3cw or c3w
cc

Yanks from cursor up to first occurrence of string case in reverse direction
Changes from cursor to beginning of line
Changes three words
Changes current line

These operators also take a repeat factor. Table 5.5 lists the various ways these operators
are used in combination with other Command Mode commands.
An operator, when doubled by itself (like dd, yy, and cc), acts only on the current line.
Note

5.9.1

Deleting and Moving Text (d, p, and P)
We used x and X for deletion; now let’s see how deletion is performed with operators.
Let’s assume that you have to delete text from the present cursor position to the end of
the line. You can frame the command yourself if you recall that the $ takes the cursor
to end of a line. The answer is simple:
d$

Deletes rest of line

d$ is a combination of an operator and a navigation command. w moves forward
one word, and G takes the cursor to the end of file, so we can confidently use these
commands:
dw Deletes one word
3dw Deletes three words
dG Deletes from current cursor position to end of file
Note how we could easily devise specialized functions by following a simple set of rules.
The operator-command theory, however, takes a backseat when deleting lines. Entire
lines are removed with the dd “command” (rather a doubled operator), which can also
be used with a repeat factor. Move the cursor to any line and then press:
dd Deletes a single line
6dd Deletes current line and five lines below

das76205_Ch05_122-155.indd 142

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

143

FIGURE 5.18

Deleting and Moving Text with the d Operator and p

Original Text
case $# in

vi Commands
#Check arguments

Transformed Text

d$

case $# in

echo “Enter
E
the filename\c”

2dw

echo “filename\c”
f

close(fd1);
(
close(fd2);

dd

cclose(fd2);

if (access(“foo”, R_OK) == -1)
printf(“Not readable “);
if (access(“foo”, F_OK) == -1)
quit(“File not found”, 1);

2dd

iif (access(“foo”, F_OK) == -1)
quit(“File not found”, 1);

p

if (access(“foo”, F_OK) == -1)
quit(“File not found”, 1);
iif (access(“foo”, R_OK) == -1)
printf(“Not readable “);

Now put the two deleted lines here:
if (access(“foo”, F_OK) == -1)
quit(“File
e not found”, 1);

How do we move text? Use the p and P commands in the same way we used them with
x and X (5.7.1) except that there are four possibilities this time:
• When we delete entire lines, we have to think in terms of “below” and “above” the
current line. p places text below the current line, and P places text above.
• When we delete a partial line, say, a word with dw, we can put the deleted word
only to the left (with p) or right (with P) of the cursor position, and not above or
below the current line.
Fig. 5.18 illustrates the use of the d operator for line deletion and movement.
5.9.2

Yanking Text (y, p and P)
The y operator yanks (or copies) text. It is used in the same way the d operator is combined with another command or used by itself. You can yank a word, a group of words,
line segments, or even entire lines with this operator. Moreover, the p and P commands
act in the same way for putting the copied text at its destination.
For instance, to yank five lines of text, move the cursor to the first of these lines
and press
5yy

Next, move the cursor to the new location, and press
p or P

to put the copied text below the current line. Adopting the same logic used for the d
operator, one can say that y$ yanks text from current position to end of line, and y1G

das76205_Ch05_122-155.indd 143

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

144
FIGURE 5.19

Copying Lines with the y Operator and p

Original Text

vi Commands

Transformed Text

close(fd1);
e

yy

close(fd1);
e
(No change)

e
close(fd1);

p

close(fd1);
close(fd1);
c

FIGURE 5.20

Changing Text with the c Operator

Original Text

vi Commands

Transformed Text

bi
#!/bin/sh

c$usr/bin/bash[Esc]

#!/usr/bin/bashh

(S
fprintf(STDOUT,
“Error\n”);

cwstderr[Esc]

r “Error\n”);
fprintf(stderr,

if grep stderr ffoo foo1 foo2

3cwfoo*[Esc]

if grep stderr foo**

(or 1yG) copies text from current cursor position to the beginning of the buffer. See for
yourself where p places the copied text—right or below. Copying lines is illustrated in
Fig. 5.19.
5.9.3

5.10

Changing Text (c)
The c operator changes text, but in the Input Mode. cw changes a word, and c$ changes
text to the end of the line. This time, the boundary limiting your area of operation could
be indicated by a $, but you may not see the $ on all systems.
If you change three words with 3cw, a $ may appear at the end of the third word
(vim excepted). The inserted text overwrites the characters delimited by the $ that
temporarily appears on the screen. If the replaced text is larger, then once the cursor
moves up to the $ marker, further insertion shifts existing text to the right. You must use
[Esc] to terminate the change operation. Changing text is shown in Fig. 5.20.

Copying and Moving Text from One File to Another
You can perform cut-and-paste and copy-and-paste operations between two files using
modified forms of the preceding techniques. You need to be familiar with these additional features:
• vi uses the :e foo command to switch to another file. But this operation will only
work if the current file is saved.
• You can toggle between the current and the previous file using either :e# or
[Ctrl-^].
• To copy or move text between two files, the standard buffer that we restore from
using p or P won’t do. We have to use a special buffer that can have any letter as
its name. The buffer a is accessed with “a. So, to copy a line to this buffer, we
need to use “ayy.

das76205_Ch05_122-155.indd 144

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

145

Space constraints don’t permit discussion of these features in this edition, but this
knowledge is adequate to copy or move a block of text from one file to another. Just
follow these steps for moving text:
1. Delete the text into a buffer a. If you are deleting four lines, then use “a4dd, where
the normal delete command is preceded by the string “a.
2. Save the current file with :w.
3. Open a new file using the ex Mode command :e foo.
4. Navigate to the desired location, press “, the buffer name (a) and p to place the
copied text below the current line: “ap (the normal put command preceded by “a).
5. You can now go back to the previous file using :e# or [Ctrl-^].
To copy text, replace the delete command with the yank command. To copy four words
from one file to another, just use 4yw instead of 4dd; everything else remains the same,
except that you don’t need to save the original file this time. The filling up of a buffer
named a is a special case of a general vi feature—the ability to copy or move up to 26
sections of text.

Tip

Linux

5.11

The sequence :e# or [Ctrl-^] may not work if the current buffer is unsaved. For this to work at
all times, make the ex Mode setting :set autowrite or :set aw. This ensures that the file is
saved automatically before the switch occurs.

To copy or move a block of text, vim doesn’t need to use the “a symbols at all. Just
delete or copy the text, save the file with :w if necessary, switch to the next file with
:e foo, and then paste the text with p. vim requires the buffer symbols only when
multiple sections are copied or moved.

Undoing Last Editing Instructions (u and U)
vi’s most notable weakness is its undo feature, which uses the u and U commands. Unlike
the Windows-based word processors, the u command permits only single-level undoing.
You can only reverse your previous editing instruction by pressing
u

Must use in Command Mode; press [Esc] if necessary

Another u at this stage will undo this too, i.e., restore the original status. So, if you have
used 10dd at the wrong location, then before you do anything else, just press u. The
command undoes changes made in the Input Mode also. If you have wrongly inserted
a block of text, press [Esc] and then u.
When a number of editing changes have been made to a single line, vi allows
you to discard all of the changes before you move away from the line. The command
U

Don’t move away from current line

reverses all changes made to the current line, i.e., all modifications that have been made
since the cursor was moved to this line.

das76205_Ch05_122-155.indd 145

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

146

Make sure the cursor has not been moved to another line before invoking U; otherwise it
won’t work.
Caution

Linux

5.12

Multiple Undoing and Redoing vim supports multilevel undoing. u behaves
differently here; repeated use of this key progressively undoes your previous actions.
You could even have the original file in front of you! Further, 10u reverses your last
10 editing actions. The function of U remains the same.
You may overshoot the desired mark when you keep u pressed, in which case
use [Ctrl-r] to redo your undone actions. Further, undoing with 10u can be completely
reversed with 10[Ctrl-r]. The undoing limit is set by the ex Mode command
:set undolevels=n, where n is set to 1000 by default.

Searching for a Pattern (/ and ?)
vi is extremely strong in search and replacement activities. Searching can be made in
both forward and reverse directions and can be repeated. It is initiated from the Command
Mode by pressing a /, which shows up in the last line. For example, if you are looking
for the string printf, enter this string after the /:
/printf[Enter]

Searches forward

The search begins forward to position the cursor on the first instance of the word. vi
searches the entire file, so if the pattern can’t be located until the end of file is reached,
the search wraps around to resume from the beginning of the file. If the search still fails,
vi responds with the message Pattern not found.
Likewise, the sequence
?pattern[Enter]

searches backward for the previous instance of the pattern. The wraparound feature also
applies here but in the reverse manner.
5.12.1

Repeating the Last Pattern Search (n and N)
The n and N commands repeat a search where n and N don’t exactly play the roles you’d expect
them to. For repeating a search in the direction the previous search was made with / or ?, use
n

Repeats search in same direction of original search

The cursor will be positioned at the beginning of the pattern. You can then carry out
some editing function, say, change a word with cw. You can press n repeatedly to scan
all instances of the string and then repeat the change wherever you want.
N reverses the direction pursued by n, which means you can use it to retrace your
search path. The search and repeat actions are illustrated in Fig. 5.21, and the commands
are summarized in Table 5.6.
What makes searching in vi powerful is that the search pattern is not confined
to a simple string. You can also use a regular expression that comprises some special

das76205_Ch05_122-155.indd 146

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

147
Search and Repeat with / and n

FIGURE 5.21

if [ -z "$pname" ]

/echo [Enter]

then
echo "You have not entered the string"
exit 1

n

else
echo "Enter the file to be used: \c"

n

read flname
if [ ! -n "$flname" ] ; then
echo "You have not entered the filename" ; exit 2

N
else
TA B L E 5 . 6

Search and Replace Commands

Command

Function

/pat
?pat
n

Searches forward for pattern pat
Searches backward for pattern pat
Repeats search in same direction along which previous search
was made (no repeat factor)
Repeats search in direction opposite to that along which previous
search was made (no repeat factor)
Replaces first occurrence of string or regular expression s1 with
string s2 in lines n1 to n2
Replaces all occurrences of find with look in lines 1 to 10
Interactively replaces find with look from current line through end
Repeats last substitution on current line (Linux only)

N
:n1,n2s/s1/s2/
:1,10s/find/look/g
:.,$s/find/look/gc
:s

characters similar to the way wild cards are used for matching filenames. There’s a
separate chapter devoted almost exclusively to regular expressions (Chapter 10), and
most of what is discussed there applies to vi too.

Note

5.13

n doesn’t necessarily repeat a search in the forward direction; the direction depends on the
search command used. If you used ?printf to search in the reverse direction in the first place,
then n also follows the same direction. In that case, N will repeat the search in the forward
direction, and not n.

Repeating the Last Command (.)
Most editors don’t have the facility to repeat the last editing instruction, but vi has. The . (dot)
command is used for repeating both Input and Command Mode commands. The principle is:
Use the actual command only once, and then repeat it at other places with the dot command.

das76205_Ch05_122-155.indd 147

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

148

To take a simple example, if you have deleted two lines of text with 2dd, then to
repeat this operation elsewhere, all you have to do is to position the cursor at the desired
location and press
.

Use u to undo this repeat

This will repeat the last editing instruction performed; i.e., it will also delete two lines
of text.
The . command is indeed a very handy tool. As another example, consider that
you have to indent a group of lines by inserting a tab at the beginning of each line. You
need to use i[Tab][Esc] only once, say on the first line. You can then move to each line
in turn by hitting [Enter] and simply pressing the dot. A group of lines can be indented
in no time.

Tip

The three commands, / (search), n (repeat search), and . (repeat last editing command), form
a wonderful trio of search—search-repeat—edit-repeat commands. You’ll often be tempted
to use this trio in many situations where you want the same change to be carried out at a
number of places.
For instance, if you want to replace some occurrences of int with double, then first search
for int with /int, change int to double with cw, repeat the search with n, and press the
. wherever you want the replacement to take place. Yes, you wouldn’t like printf to also
show up (int is embedded there), which means you need to use regular expressions to throw
printf out.

5.14

Substitution—Search and Replace (:s)
vi offers yet another powerful feature, that of substitution, which is achieved with the
ex Mode’s s (substitute) command. It lets you replace a pattern in the file with something
else. The / serves as the delimiter of the syntax components:
:address/source_pattern/target_pattern/flags

The source_pattern here is replaced with target_pattern in all lines specified by address.
The address can be one or a pair of numbers, separated by a comma. For instance, 1,$
addresses all lines in a file. The most commonly used flag is g, which carries out the
substitution for all occurrences of the pattern in a line. The following example shows a
typical substitution command:
:1,$s/double/float/g

Can also use % instead of 1,$

Here, double is replaced with float globally throughout the file. If you leave
out the g, then the substitution will be carried out for the first occurrence in each
addressed line.
The target pattern is optional. If you leave it out, then you’ll delete all instances
of the source pattern in all lines matched by the address:
:1,50s/unsigned//g

das76205_Ch05_122-155.indd 148

Deletes unsigned everywhere in lines 1 to 50

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
FIGURE 5.22

149
Interactive Substitution with :s

message=”has scheduled the $1 command”
^^^^^^y
e_message=”are not using the $1 command”
^^^^^^n
if grep “$1” $crondir/henry ; then
echo “henry $message”
^^^^^^y
elif grep “$1” $crondir/julie ; then
echo “julie $message”
^^^^^^y
else
echo “The three users $e_message”
^^^^^^n
fi
Before Substitution

msg=”has scheduled the $1 command”
e_message=”are not using the $1 command”
if grep “$1” $crondir/henry ; then
echo “henry $msg”
elif grep “$1” $crondir/julie ; then
echo “julie $msg”
else
echo “The three users $e_message”
fi

After Substitution

As shown above, you can choose the range of lines that are to be affected by the
substitution. The following examples should make addressing clear:
:3,10s/msg/message/g
:$s/msg/message/g
:.s/echo/printf/

All occurrences in lines 3 through 10
All occurrences in last line
Only first occurrence in current line

Interactive Substitution Sometimes you may want to selectively replace a string. In
that case, add the c (confirmatory) parameter as the flag at the end:
:1,$s/message/msg/gc

Each line is selected in turn, followed by a sequence of carets in the next line, just below
the pattern that requires substitution (Fig. 5.22). A y performs the substitution; any other
response doesn’t. This sequence is repeated for each of the matched lines in turn. In the
present case, the substitution is performed for only three of the five lines.

Linux

The interactive substitution feature in vim is both friendlier and more powerful than its
UNIX counterpart. The string to be replaced is shown in reverse video, and a prompt
appears in the last line of the screen:
replace with fprintf (y/n/a/q/^E/^Y)?

Apart from responding with y or n, you have the option of aborting (q) the substitution
process or making it noninteractive (a). [Ctrl-e] and [Ctrl-y] are meant for scrolling.

5.15 set: Customizing vi
vi can be tailored by redefining keys or abbreviating frequently used strings, but in
this section we examine some of the variable settings that benefit the programmer.

das76205_Ch05_122-155.indd 149

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

150
TA B L E 5 . 7

set Options

Option

Abbreviation

Significance

autoindent
autowrite

ai
aw

ignorecase
magic

ic
-

number
showmatch

nu
sm

Next line starts at previous indented level
Writes current file automatically whenever switching
files with :e
Ignores case when searching for patterns
Treats regular expression characters as special when
searching for patterns
Displays line numbers on screen
Shows momentarily match to a ) and }

showmode
tabstop
wrapscan

ts
ws

Displays a message when vi is in Input Mode
Sets tabs for display (default: eight spaces)
Continues pattern search by moving to other end of
a file so that entire file is scanned

These settings (Table 5.7) can be made at the ex Mode prompt, but they become
permanent only when they are placed in ~/.exrc (or ~/.vimrc in Linux).
The :set command is used to set vi variables, and we have used some of them
already (like :set showmode, :set autowrite, etc.). Many of these variables can
have the string no prefixed to their names, in which case the setting is deactivated. For
instance, noautowrite (or noaw) negates autowrite (or aw). Other variables are of the
form variable=value. Let’s take a look at some of these variables.
Automatic Indentation (autoindent) Programmers need to provide indentation to
their code for easier readability. Nested if statements along with their corresponding
fis should appear at the right indentation. This aspect is taken care of when you use
the following set statement:
:set autoindent

Or :set ai

When this option is set, an [Enter] in the Input Mode places the cursor in the next line
at the current indentation.
Numbering Lines (number) The number setting shows all lines duly numbered. This
makes it easier to debug programs. To reverse this setting (i.e., to switch back to the
nonumber option), you simply have to prefix the string no to the option:
:set nonumber

Ignoring Case in Pattern Searches (ignorecase) The search commands (not the
substitution ones) may or may not pursue a case-insensitive search. That depends on
the ignorecase setting. By default, this is generally off, but you can turn it on with
:set ignorecase.

das76205_Ch05_122-155.indd 150

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

151

See Matching } or ) (showmatch) Another option, especially useful for C, Java, and
perl programmers, enables the programmer to see momentarily the matching bracket
to a ) or }. The showmatch option helps you locate matching brackets:
:set showmatch

When text is entered in the Input Mode, the moment a ) or } is entered, the cursor will
jump to its matching counterpart and stay there for a fraction of a second before returning
to its current location. If a match is not found, the system responds with a beep.
Setting Tab Stops (tabstop=n) Too many nested programming constructs often cause
a line to wrap around. To some extent, this can be alleviated by changing the default tab
setting (eight spaces). You can reduce this value, say, to four, with :set tabstop=4.
All settings made with set are displayed with the :set all command. If you find
these settings useful, then save them in ~/.exrc (~/.vimrc in vim). vi reads this file only
once, upon startup, so you have to restart vi after you have modified the configuration file.
➤

GOING FURTHER

The map command lets you assign a set of keystrokes to a key. Take, for instance, the
key sequence :w[Enter] that saves your buffer. You can map the key g to this sequence
by using map in the ex Mode:
:map g :w^M

GOING FURTHER

5.16 map: Mapping Keys of Keyboard

^M signifies the [Enter] key

This mapping also includes the [Enter] key, which vi understands as [Ctrl-m] (shown
as ^M). This character is entered by first pressing [Ctrl-v] and then [Ctrl-m] (5.3.5). You
can now press g in the Command Mode to save your buffer. To map a function key, say
[F1], use #1 to signify the key. On some systems like Solaris, you’ll have to first press
[Ctrl-v] and then the actual function key itself.
You can also map keys in the Input Mode using the ! as suffix to map. This sequence
maps the function key [F2] in the Input Mode:
:map! #2 ^[:w^M

Function key [F2] is #2

The string ^[ is actually the [Esc] character which switches vi to the Command Mode
before :w saves the buffer.
The :map command displays the mapped environment. :unmap cancels a Command
Mode map, and :unmap! cancels an Input Mode map.

Tip

Did you know that you can compile your C program or execute a shell or perl script from
inside the editor with a single keystroke? vi understands the current file as %, so cc % compiles
the current file. Let’s map the function key [F3] to invoke this function:

:map #3 :!cc %^M

das76205_Ch05_122-155.indd 151

[F3] now compiles current C program

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

152

The ex Mode command :! invokes a temporary shell escape (Table 5.2) which is used to run cc
(the C compiler). To invoke a shell or perl script, make this mapping of the function key [F4]:

:map #4 :!%^M

[F4] now executes current script

These are two important mappings used by this author. Note that scripts need to have execute
permission, so you should first use :!chmod 755 % to assign this permission. You don’t have
to leave vi for this task either.

5.17 abbr: Abbreviating Text Input

GOING FURTHER

The abbreviate command (itself abbreviated to ab) is used to expand short strings
to long words. The syntax is similar to map except that the string that is mapped can
consist of more than one character. Here are some important abbreviations that benefit
C and Java programmers:
:ab
:ab
:ab
:ab

pf printf
incstd #include 
sopl System.out.println
psvm public static void main (String args[ ])

To consider the first example, when you enter the word pf, followed by a key
which is neither alphanumeric nor _ (underscore), pf gets expanded to printf.
Store all sets, maps, and abbreviations in $HOME/.exrc ($HOME/.vimrc for vim).
Carry this file with you wherever you go; it could be your most precious possession.
The other features of vi are taken up in Appendix D.

SUMMARY
vi operates in three modes. The Command Mode is used to enter commands that operate
on text or control cursor motion. The Input Mode is used to enter text. The ex Mode (or
Last Line Mode) is used for file handling and substitution.
Most of the Input and Command Mode commands also work with a repeat factor,
which generally performs the command multiple times.
The Input Mode is used to insert (i and I), append (a and A), replace (r and R), and
change (s or S) text and to open a line (o and O). The mode is terminated by pressing [Esc].
You can enter control characters (using [Ctrl-v] first) and then the character. The
[Esc] character is inserted using [Ctrl-v][Ctrl-[].
The ex Mode is invoked by pressing a : in the Command Mode. You can save your
work (:w), exit the editor after saving (:x), and quit without saving (:q and :q!). You
can write selected lines to a separate file by using line addresses with :w. Sometimes
you’ll need to escape to a shell (:sh) without quitting the editor.
Navigation is performed in the Command Mode. You can move in the four directions
(h, j, k, and l) or move along a line, using a word as a navigation unit. You can move
back (b) and forward (w) to the beginning of a word. The beginning of a line is reached

das76205_Ch05_122-155.indd 152

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor

153

with 0 and the end of line with $. You can know your current line number ([Ctrl-g]) and
go to a specific line number (G). You can use the control keys to page forward and back.
You can delete characters (x and X) without using operators. Deleted text can be
put at another location (p and P).
vi’s editing power lies in its operators. By combining an operator with a Command
Mode command, you can delete (d), yank (y), and change (c) text in practically any
manner. When the operator is used doubly, the operation affects the current line only.
vi can repeat (.) and undo (u) the last editing instruction. You can undo all changes
made to the current line (U). vim in Linux can perform multiple levels of undo and redo
with u and [Ctrl-r], respectively.
You can search for a pattern (/ and ?) and repeat (n and N) the search in both directions. The /, n and . commands form a very useful trio for interactive replacement work.
The ex Mode is also used for substitution (:s). Substitution can be global (g flag)
or confirmatory (c flag). Both search and replace operations also use regular expressions
for matching multiple patterns.
With the :set command, lines can be numbered (number) and automatically indented (autoindent). You can ensure that a file is saved before switching (autowrite).
Searches can be made case-insensitive (ignorecase). The no prefix reverses a setting.
All :set commands should be placed in $HOME/.exrc (or $HOME/.vimrc for
vim) so they are always available on startup.

SELF-TEST
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
5.11
5.12
5.13

das76205_Ch05_122-155.indd 153

How will you insert a line (i) above the current line, (ii) below the current line?
How will you (i) replace has with have in the current line, (ii) change the current
line completely?
Your screen shows junk. How do you clear it?
Name three ways of exiting a vi session after saving your work. How do you
abort an editing session?
You pressed 50k to move the cursor 50 lines up, but you see 50k input as text.
What mistake did you make, and how do you remove the three characters?
In the current line, how do you take your cursor to the (i) 40th character,
(ii) beginning, (iii) end?
Name the commands required to move quickly to the fifth word of a line and
replace its four characters with the string counter.
Find out the number of words in this string as interpreted by (i) vi and
(ii) wc—29.02.2000 is_last_day_of_February.
Explain which of the following commands can be repeated or undone: (i) 40k,
(ii) [Ctrl-f], (iii) 5x, (iv) J.
You have incorrectly entered the word Comptuer. How will you correct it to
Computer?
Five contiguous lines contain only lowercase letters. How do you combine them
into a single line and then convert the entire line to uppercase?
How will you compile a C program without leaving the editor?
What is the significance of the n and N commands?

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

154
5.14
5.15
5.16
5.17
5.18
5.19
5.20
5.21
5.22

Every time you press a . (dot), you see a blank line inserted below your current
line. Why does that happen?
How do you save the current line to a separate file? What do you do if the file exists?
How do you (i) delete text from the current line to the beginning of the file,
(ii) copy 10 characters, (iii) copy 10 words?
How will you copy five lines of text from one file to another? How do you then
toggle between the two files?
How do u and U differ? When will U fail to work?
Fundamentally, how are d and y different from Command Mode commands like
j and $?
How do you noninteractively and globally replace Internet with Web in all lines
of a file?
How will you revert to the last-saved version of a file?
How will you ensure that vi automatically saves a file when moving from one
file to another?

EXERCISES
5.1
5.2
5.3
5.4
5.5

5.6

5.7

5.8

5.9

5.10
5.11
5.12

das76205_Ch05_122-155.indd 154

Name the three modes of vi, and explain how you can switch from one mode
to another.
How will you add /* at the beginning of a line and */ at the end?
How do you remove the characters that you inserted in Exercise 5.2?
vi refuses to quit with :q. What does that indicate, and how do you exit anyway?
Explain what the following commands do: (i) :.,10w foo, (ii) :$w! foo. In
which mode are the commands executed, and what difference does it make if
foo already exists?
In the midst of your work, how can you see the list of users logged in? If you
have a number of UNIX commands to execute, which course of action will you
take?
Name the commands required to move to the line containing the string #include,
delete four lines there, and then place the deleted lines at the beginning of the
file.
Name the commands required to replace (i) echo ‘Filename: \c’ with
echo -n “Filename: “ (ii) printf(“File not found\n); with
fprintf(stderr, “File not found\n”); (iii) echo “1234567890” with
echo “12345678901234567890”.
How do you copy text from the current cursor position to the following: (i) the
character under the cursor, (ii) the beginning of line, (iii) 10 words, (iv) following
text up to the string esac.
What commands will you use to delete (i) text from the current cursor position
to end of file, (ii) entire contents of file?
How do you move to line number 100 and then write the remaining lines (including that line) to a separate file?
Name the commands required to interactively replace printf( with
fprintf(stderr,. How will you repeat the action globally?

12/13/11 10:44 AM

Chapter 5: The vi/vim Editor
5.13
5.14

5.15

5.16
5.17
5.18
5.19
5.20
5.21
5.22
5.23
5.24

155

How will you search for a pattern printf and then repeat the search in the
opposite direction from that in which the original search was made?
Name the commands required to noninteractively replace all occurrences of cnt
with count in (i) the first 10 lines, (ii) the current line, (iii) all lines. How do you
repeat the exercise in an interactive manner?
Name the commands required to delete text delimited by { and } where both
characters occur after the current cursor position. HINT: Follow the pattern search
with a terminating / and use a + after the /.
If the power to the machine is cut off while a vi session is active, how does it
affect your work? What salvage operation will you try?
You copied 20 lines with 20yy, then switched to another file with :e foo, but
when you tried to paste these lines back with p, it didn’t work. Why?
You made some changes to a read-only file and then found that you can’t save
the buffer. What course of action will you take without quitting the editor?
You created a shell script with vi. How will you make the file executable and
then execute the script without leaving the editor?
Frame a command to change text from the current position to the first occurrence
of the pattern Packet Switching.
You need to shift the first five lines to the right by two spaces. How do you do
that (i) interactively, (ii) noninteractively?
How will you ensure that vi (i) automatically saves a file when moving from
one file to another, (ii) expands each tab to three spaces?
Copy /etc/passwd to passwd. Name the vi commands required to save the first
10 lines in passwd1, the next 10 in passwd2, and the rest in passwd3.
List the editing and navigation commands required to convert the following text:
# include
void quit (char *message)
{
printf(“Error encountered\n”);
printf(“error number %d, “, errno);
printf(“quitting program\n”);
exit(1);
}
to this:
#include 
#include 
void quit (char *message, int exit_status) {
/* printf(“Error encountered\n”); */
fprintf(stderr, “Error number %d, quitting program\n”, errno);
exit(exit_status);
}

das76205_Ch05_122-155.indd 155

12/13/11 10:44 AM

CHAPTER

6

The Shell

T

his chapter introduces the agency that sits between the user and the UNIX
system. It is called the shell. All of the wonderful things that we can do with
UNIX are possible because the shell does a lot of work on our behalf that could be tedious for us to do on our own. The shell looks for special symbols in the command line,
performs the tasks associated with them, and finally executes the command. For example,
it opens a file to save command output whenever it sees the > symbol.
The shell is both a command interpreter and a programming language. It is also
a process that creates an environment to work in. All of these features deserve separate
chapters for discussion, and you’ll find the shell discussed at a number of places in this
book. In this chapter, we focus on the shell’s basic interpretive activities. We have seen
some of these activities in previous chapters (like rm * or ls | more), but it is here
that we need to examine them closely.

Objectives
• Gain an overview of the shell’s interpretive cycle.
• Learn the significance of metacharacters and their use in wild cards for matching
multiple filenames.
• Know the use of escaping and quoting to remove the meaning of a metacharacter.
• Learn the difference between the use of double and single quotes.
• Discover the significance of the three standard files (streams) that are available to
every command.
• Learn how the shell manipulates the default source and destination of these streams
to implement redirection and pipelines.
• Understand what filters are and why they are so important in UNIX.
• Learn the significance of the files /dev/null and /dev/tty.
• Know the importance of making the command ignorant of the source of its input and
destination of its output.
• Know the use of command substitution to obtain the arguments of a command from
the standard output of another.
• Understand shell variables and why they are so useful.

156

das76205_Ch06_156-187.indd 156

12/13/11 10:44 AM

157

Chapter 6: The Shell

• Learn additional wild cards used by the Korn shell and Bash (Going Further).
• Learn about passing a list of filenames down a pipeline for use by xargs (Going
Further).

6.1

The Shell as Command Processor
When you log on to a UNIX machine, you first see a prompt. Even though it may appear
that nothing is happening there, a UNIX command is in fact running at the terminal. But
this command is special; it starts functioning the moment you log in and withers away
when you log out. This command is the shell. If you run the ps command (that shows
processes), you’ll see it running:
$ ps
PID TTY
526 pts/6

TIME CMD
0:00 bash

Bash shell running

When you key in a command, it goes as input to the shell. The shell first scans the
command line for metacharacters. These are special characters that mean nothing to
the command, but mean something special to the shell. The previous example (ps) had
none of these characters, but we did encounter these characters before (section numbers
shown in parentheses):
echo date > date.sh
rm *
ls | more

(4.2)
(3.14)
(3.16)

When the shell sees metacharacters like the >, |, *, etc. in its input, it translates these
symbols to their respective actions before the command is executed. It replaces the * with
almost all filenames in the current directory (but skips hidden files) so that rm ultimately
runs with these names as arguments. On seeing the >, the shell opens the file date.sh
and connects echo’s output to it. The shell re-creates the command line by removing all
metacharacters and finally passes on the command to the kernel for execution.
Note that the shell has to interpret these metacharacters because they usually mean
nothing to the command. To sum up, the following activities are typically performed by
the shell in its interpretive cycle:
• It issues the prompt and waits for you to enter a command.
• After a command is entered, the shell scans the command line for metacharacters and
expands abbreviations (like the * in rm *) to re-create a simplified command line.
• It then passes on the command line to the kernel for execution.
• The shell waits for the command to complete and normally can’t do any work
while the command is running.
• After command execution is complete, the prompt reappears and the shell returns to
its waiting role to start the next cycle. You are now free to enter another command.
You can change this default behavior and instruct the shell not to wait so you can run
multiple commands. We’ll examine the technique of doing that in Chapter 7.

das76205_Ch06_156-187.indd 157

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

158

6.2 Shell Offerings
Your UNIX system offers a variety of shells for you to choose from. Over time, shells
have become more powerful by the progressive addition of new features. The shells we
consider in this text can be grouped into two categories:
• The Bourne family comprising the Bourne shell (/bin/sh) and its derivatives—the
Korn shell (/bin/ksh) and Bash (/bin/bash).
• The C Shell (/bin/csh) and its derivative, Tcsh (/bin/tcsh).
To know the shell you are using, invoke the command echo $SHELL. The output could be
one of the absolute pathnames of the shell’s command file shown in parentheses above.
In this chapter, we discuss the common features of the Bourne family. Korn and
Bash are supersets of Bourne, so anything that applies to Bourne also applies to them. A
few of them don’t apply to the C shell and are noted as and when they are encountered.
You may not want to know all this now, but it does pay to know the shell you are using
at this stage.

6.3

Pattern Matching—The Wild Cards
In previous chapters, you used commands with more than one filename as arguments
(e.g., cp chap01 chap02 chap03 progs). Often, you’ll need to use a command with
similar filenames:
ls chap chap01 chap02 chap03 chap04 chapx chapy chapz

The common substring here is chap, and the shell offers a facility to represent these
filenames with a single pattern. For instance, chap* represents all filenames beginning
with chap. You can use this pattern as an argument to a command rather than supply
a long list of filenames which the pattern represents. The shell will expand it suitably
before the command is executed.
The metacharacters used to match filenames belong to a category called wild
cards (something like the joker that can match any card). In the following sections,
we’ll discuss the significance of the various metacharacters in the wild-card set that are
listed in Table 6.1.
6.3.1

The * and ?
The * Now let’s get into the specifics. The * (a metacharacter) is one of the characters
of the shell’s special set, and we have used it before (rm * in Section 3.14). This character
matches any number of characters including none. When it is appended to the string
chap, the pattern chap* matches filenames beginning with the string chap—including
the file chap. You can now use this pattern as an argument to ls:
$ ls -x chap*
chap chap01 chap02 chap03 chap04 chap15 chap16 chap17 chapx chapy
chapz

das76205_Ch06_156-187.indd 158

12/13/11 10:44 AM

159

Chapter 6: The Shell
TA B L E 6 . 1

The Shell’s Wild Cards and Application

Wild Card

Matches

*
?
[ijk]
[x-z]

Any number of characters including none
A single character
A single character—either an i, j, or k
A single character that is within the ASCII range of the
characters x and z
A single character that is not an i, j, or k (Not in C shell)
A single character that is not within the ASCII range of the
characters x and z (Not in C shell)
pat1, pat2, etc. (Not in Bourne Shell; see Going Further)
All except flname (Korn and Bash; see Going Further)
All except fname1 and fname2 (Korn and Bash; see Going Further)

[!ijk]
[!x-z]
{pat1,pat2...}
!(flname)
!(fname1|fname2)
Examples
Command

Significance

ls *.lst
mv * ../bin
gzip .?*.?*

Lists all files with extension .lst
Moves all files to bin subdirectory of parent directory
Compresses all files beginning with a dot, followed by one or
more characters, then a second dot followed by one or more
characters
Copies file chap to file chap* (* loses meaning here)
Copies to progs directory all six-character filenames
Compares files rep1 and rep2
Removes files note00, note01 . . . through note19
Prints all files having extensions except C object files
Copies to the parent directory files having extensions with at
least one character before the dot, but not having 1, 2, 3, or 8 as
the last character

cp chap chap*
cp ?????? progs
cmp rep[12]
rm note[0-1][0-9]
lp *.[!o]
cp ?*.*[!1238] ..

When the shell encounters this command line, it immediately identifies the * as a
metacharacter. It then creates a list of files from the current directory that match this
pattern. It reconstructs the command line as follows, and then hands it over to the kernel
for execution:
ls -x chap chap01 chap02 chap03 chap04 chap15 chap16 chap17 chapx chapy chapz

What happens when you use echo with the * as argument?
$ echo *
array.pl back.sh calendar cent2fah.pl chap chap01 chap02 chap03 chap04 chap15 ch
ap16 chap17 chapx chapy chapz count.pl date_array.pl dept.lst desig.lst n2words.
pl name.pl name2.pl odfile operator.pl profile.sam rdbnew.lst rep1.pl

das76205_Ch06_156-187.indd 159

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

160

You simply see a list of files! All filenames (except those beginning with a dot) in the
current directory match a solitary *, so you see all of them in the output. If you use
rm * in this directory, all of these files will be deleted.

Note

Windows users may be surprised to know that the * may occur anywhere in the pattern and
not merely at the end. Thus, *chap* matches all the following filenames—chap newchap chap03
chap03.txt.
Be careful when you use the * with rm to remove files. You could land yourself in a real mess
if, instead of typing rm *.o, which removes all the C object files, you inadvertently introduce
a space between * and .o:

Caution

$ rm * .o
rm: .o: No such file or directory

Very dangerous!

The error message here masks a disaster that has just occurred; rm has removed all files in this
directory! Whenever you use a * with rm, you should pause and check the command line before
you finally press [Enter]. A safer bet would be to use rm -i.

The ? The ? matches a single character. When used with the same string chap (as chap?),
the shell matches all five-character filenames beginning with chap. Place another ? at
the end of this string, and you have the pattern chap??. Use both of these expressions
separately, and the meaning of the ? becomes obvious:
$ ls -x chap?
chapx chapy chapz
$ ls -x chap??
chap01 chap02 chap03 chap04 chap15 chap16 chap17

Both the * and ? operate with some restrictions that are examined in Section 6.3.4.
6.3.2

The Character Class
The patterns framed in the preceding examples are not very restrictive. With the knowledge we have, it’s not easy to list only chapy and chapz. Nor is it easy to match only
the first four chapters from the numbered list. You can frame more restrictive patterns
with the character class.
The character class comprises a set of characters enclosed by the rectangular
brackets [ and ], but it matches a single character in the class. The pattern [abcd] is
a character class, and it matches a single character—an a, b, c, or d. This pattern can
be combined with any string or another wild-card expression so that selecting chap01,
chap02, and chap04 now becomes a simple matter:
$ ls chap0[124]
chap01 chap02 chap04

Range specification is also possible inside the class with a - (hyphen); the two characters
on either side of it form the range of the characters to be matched. Here are two examples:

das76205_Ch06_156-187.indd 160

12/13/11 10:44 AM

161

Chapter 6: The Shell
ls chap0[1-4]
ls chap[x-z]

Lists chap01, chap02, chap03 and chap04
Lists chapx, chapy and chapz

A valid range specification requires that the character on the left have a lower ASCII
value than the one on the right.

Note

The expression [a-zA-Z]* matches all filenames beginning with a letter, irrespective of case.
You can match a word character by including numerals and the underscore character as well:
[a-zA-Z0-9_].

Negating the Character Class (!) The solution that we prescribe here unfortunately
doesn’t work with the C shell, but with the other shells, you can use the ! as the first
character in the class to negate the class. The two examples that follow should make
this point amply clear:
*.[!co]
[!a-zA-Z]*

Matches all filenames with a single-character extension but
not the .c or .o files.
Matches all filenames that don’t begin with a letter.

Even though the character class is meant to be used with a group of characters, it’s the
only way you can negate a match for a single character. Thus, to match all files with
single-character extensions except those ending with .o (i.e., C object files), you have
to use a character class as a “dummy class”:
*.[!o]

Not the .o files

The ! can’t be used to negate a character class at all. In fact, the C shell has no mechanism
for doing so.
C Shell

The Mystery of the find Command
It’s true that a command doesn’t perform wild-card expansion on its own and runs
only after the shell has expanded the wild cards. The find command is an exception. It supports wild cards (probably the only UNIX command having this feature)
in the filename that’s used as a parameter to the -name option:
find / -name “*.[hH][tT][mM][lL]” -print
find . -name “note??” -print

All .html and .HTML files
Two characters after note

Here, we are using the same wild-card characters, but this time they are a feature
of find and not of the shell. find supports only the Bourne shell set. By providing
quotes around the pattern, we ensured that the shell can’t even interpret this pattern.
In fact, allowing the shell to do that could be disastrous! You’ll learn about this
insulating feature shortly.

das76205_Ch06_156-187.indd 161

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

162
6.3.3

Matching the Dot
The behavior of the * and ? in relation to the dot isn’t as straightforward as it may seem.
The * and ? don’t match
• all filenames beginning with a . (dot).
• the / of a pathname.
If you want to list all hidden filenames in your directory having at least three characters
after the dot, then the dot must be matched explicitly:
$ ls .???*
.bash_profile

.exrc

.netscape .profile

However, if the filename contains a dot anywhere but at the beginning, it need not be
matched explicitly. For example, the expression *c also matches all C programs:
$ ls *c
access.c
check_permissions.c
runc
shell.c
toc

Note

6.3.4

Note

There are two things that the * and ? can’t match. First, they don’t match a filename beginning
with a dot, although they can match any number of embedded dots. For instance, apache*gz
matches apache_1.3.20.tar.gz. Second, these characters don’t match the / in a pathname.
You can’t use cd /usr*local to switch to /usr/local.

Rounding Up
Some of the wild-card characters have different meanings depending on where they are
placed in the pattern. The * and ? lose their meaning when used inside the class, and
they are matched literally. Similarly, - and ! also lose their significance when placed at
other locations. There are other restrictions, but we can ignore them for now.
What if the shell fails to match a single file with the expression chap*? There’s
a surprise element here; the shell also looks for a file named chap*. You should avoid
using metacharacters when choosing filenames, but if you have to handle one, then you
have to turn off the meaning of the * so that the shell treats it literally. This deactivation
feature is taken up in the next section.
The expression [!!] matches a single character filename that is not a !. This doesn’t work in
the C shell and Bash, which use the ! for a different purpose. Bash needs to use [!\!] here,
but the C shell can’t negate a character class at all.
When organizing information in files that need to be accessed as a group, choose the filenames
with care so you don’t need to use too many patterns to match them.

Tip

das76205_Ch06_156-187.indd 162

12/13/11 10:44 AM

163

Chapter 6: The Shell

6.4

Escaping and Quoting
You might well think that since the shell transforms the wild-card characters, filenames
musn’t contain these characters. That’s correct thinking, but the real world is far from
perfect. One of the examples featured in Table 6.1 shows how easy it is to actually create a file named chap* (cp chap chap*). This creates problems as the pattern chap*
also matches the filename chap*:
$ ls chap*
chap chap* chap01 chap02 chap03 chap04 chap15 chap16 chap17 chapx
chapy chapz

This file can be a great nuisance and should be removed immediately, but that won’t be
easy. rm chap* removes all of these files. We must be able to protect all special characters
(including wild cards) so the shell is not able to interpret them. The shell provides two
solutions to prevent its own interference:
• Escaping—Providing a \ (backslash) before the wild card to remove (escape) its
special meaning.
• Quoting—Enclosing the wild card, or even the entire pattern, within quotes (like
“chap*”). Anything within these quotes (barring few exceptions) is left alone by
the shell and not interpreted.
In most cases you can use either mechanism, but some situations warrant the use of
quotes. Sometimes you also need to make a distinction between single and double
quotes, but more on that later.
6.4.1

Escaping
When the \ precedes a metacharacter, its special meaning is turned off. In the pattern
\*, the \ tells the shell that the asterisk has to be treated and matched literally instead
of being interpreted as a metacharacter. Using this feature, we can now remove only
the file named chap*:
rm chap\*

Doesn’t remove chap1, chap2

This feature is known as escaping. You have seen stty using this feature (2.14), and
you’ll see other application areas also. Here’s another example:
rm chap0\[1-3\]

Removes only chap0[1-3]

Escaping the Space Apart from metacharacters, there are other characters that are
special—like the space. The shell uses the space to delimit command-line arguments.
So, to remove the file My Document.doc, which has a space embedded, a similar reasoning should be followed:
rm My\ Document.doc

das76205_Ch06_156-187.indd 163

Without the \ rm would see two files

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

164

Escaping the \ Itself Sometimes you may need to interpret the \ itself literally. You
need another \ before it, that’s all:
$ echo \\
\
$ echo The newline character is \\n
The newline character is \n

Ignoring the Newline Character Command lines that use several arguments often
overflow to the next line. To ensure better readability, split the wrapped line into two
lines, but make sure that you input a \ before you press [Enter]:
$ find /usr/local/bin /usr/bin -name “*.pl” -mtime +7 -size +1024 \[Enter]
> -size -2048 -atime +25 -print
Note the >

The \ here ignores [Enter]. It also produces the second prompt (which could be a > or
a ?), which indicates that the command line is incomplete. For better readability, you
should split long pipelines (6.8) into multiple lines wherever possible.

Note

6.4.2

The space, \, and LF (the newline character generated by [Enter]) are also special and need to
be escaped if the shell is to be prevented from interpreting them in the way it normally does.

Quoting
There’s another way to turn off the meaning of a metacharacter. When a command argument is enclosed in quotes, the meanings of all enclosed special characters are turned
off. Here’s how we can run some of the previous commands, using a mix of single- and
double-quoting this time:
echo ‘\’
rm ‘chap*’
rm “My Document.doc”

Displays a \
Removes file chap*
Removes file My Document.doc

Escaping also turns out to be a tedious affair when there are just too many characters to
protect. Quoting is often a better solution. The following example shows the protection
of four special characters using single quotes:
$ echo ‘The characters |, <, > and $ are also special’
The characters |, <, > and $ are also special

We could have used escaping here, but then we would need to use a \ in front of each
of these four metacharacters. We used single quotes because they protect all special
characters (except the single quote). Double quotes are more permissive; they don’t
protect (apart from the double quote itself) the $ and the ` (backquote):
$ echo “Command substitution uses `` while TERM is evaluated using $TERM”
Command substitution uses while TERM is evaluated using vt100

das76205_Ch06_156-187.indd 164

12/13/11 10:44 AM

165

Chapter 6: The Shell

Observe that the pair of backquote characters (``) and the variable $TERM have been interpreted by the shell inside double quotes. The value of $TERM is vt100 for this terminal,
and `` evaluated to a null command. Now try out the same example using single quotes:
$ echo ‘Command substitution uses `` while TERM is evaluated using $TERM’
Command substitution uses `` while TERM is evaluated using $TERM
It’s often crucial to select the right type of quote, so bear in mind that single quotes protect
all special characters except the single quote and \. Double quotes, however, allow a pair of
backquotes (``) to be interpreted as command substitution characters, and the $ as a variable
prefix. There is also a reciprocal relationship between the two types of quotes; double quotes
protect single quotes, and single quotes protect the double.

Note

6.4.3

Escaping in echo
We used echo in Section 2.6 with escape sequences like \n and \t. The \ has a reverse
meaning there; it treats the characters n and t as special rather than removing their
special meaning because n and t don’t have special meaning. These escape sequences
are always used within quotes to keep the shell out. But what is echo?
$ type echo
echo is a shell builtin

We have a funny situation here. We quoted echo’s arguments to keep the shell out of the
picture only to learn that echo is built into the shell! For this purpose, the shell treats
echo as an external command (which it once was).

6.5 Redirection
Before we commence our discussions on redirection, let’s first understand what the term
terminal means since we’ll be using it often. In the context of redirection, the terminal
is a generic name that represents the screen, display, or keyboard (or even an X window
that emulates a terminal). Just as we refer to a directory as a file, we’ll also sometimes
refer to the keyboard as a terminal.
We see command output and error messages on the terminal (display), and we
sometimes provide command input through the terminal (keyboard). The shell associates
three files with the terminal—two for the display and one for the keyboard. Even though
our terminal is also represented by a specific device name, commands don’t usually read
from or write to this file. They perform all terminal-related activity with the three files
that the shell makes available to every command.
These special files are actually streams of characters that many commands see as
input and output. A stream is simply a sequence of bytes. When a user logs in, the shell
makes available three files representing three streams. Each stream is associated with a
default device, and—generically speaking—this device is the terminal:
• Standard input—The file (or stream) representing input, which is connected to
the keyboard.
• Standard output—The file (or stream) representing output, which is connected
to the display.

das76205_Ch06_156-187.indd 165

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

166

• Standard error—The file (or stream) representing error messages that emanate
from the command or shell. This is also connected to the display.
A group of UNIX commands reads from and writes to these files. A command is usually
not designed to send output to the terminal but to this file. Likewise, it is not designed
to accept input from the keyboard either, but only from a standard file which it sees as a
stream. Every command that uses streams will always find these files open and available.
Even though the shell associates each of these files with a default physical device,
this association is not permanent. The shell can easily unhook a stream from its default
device and connect it to a disk file (or to any command) the moment it sees some special
characters in the command line. You, as user, have to instruct the shell to do that by
using symbols like > and < in the command line. This means that instead of input and
output coming from and to the terminal, they can be redirected to come from or go to
any disk file.
6.5.1

Standard Input
We have used the cat and wc commands to read disk files. These commands have an
additional method of taking input. When they are used without arguments, they read the
file representing the standard input. This file is indeed special; it can represent three
input sources (Fig. 6.1):
• The keyboard, the default source.
• A file using redirection with the < symbol (a metacharacter).
• Another program using a pipeline (to be taken up later).
When you use wc without an argument and have no special symbols like the < and | in
the command line, wc obtains its input from the default source. You have to provide this
input from the keyboard and mark the end of input with [Ctrl-d]:
$ wc
Standard input can be redirected
It can come from a file
or a pipeline
[Ctrl-d]
3
14
71

FIGURE 6.1

Three Sources of Standard Input

Pipe
Keyboard
(Default)
File

das76205_Ch06_156-187.indd 166

command
The quick brown fox jumbed
over the lazy dog. The quick
brown fox jumbed over the
lazy dog. The quick brown fox
jumbed over the lazy dog. The
quick brown fox jumbed over
the lazy dog.

12/13/11 10:44 AM

Chapter 6: The Shell

167

The wc command, which takes the stream from standard input, immediately counts 3
lines, 14 words, and 71 characters. Now run wc with a filename as argument:
$ wc /etc/passwd
21
45
1083 /etc/passwd

There’s a fourth column here; wc prints the filename because it opened the file itself.
In the other example, no filename was specified, so no filename was output. wc simply
read the standard input file that was opened by the shell when you logged in.
The shell’s manipulative nature is useful here. It can reassign or redirect the
standard input to originate from a file on disk. This redirection requires the < symbol:
$ wc < /etc/passwd
21
45
1083

The filename is missing once again, which means that wc didn’t open /etc/passwd. It
read the standard input file as a stream but only after the shell made a reassignment of
this stream to a disk file. The sequence works like this:
1. On seeing the <, the shell opens the disk file, /etc/passwd, for reading.
2. It unplugs the standard input file from its default source and assigns it to
/etc/passwd.
3. wc reads from standard input that has previously been reassigned by the shell to
/etc/passwd.
The important thing here is that wc has no idea where the stream came from; it is not
even aware that the shell had to open the file /etc/passwd on its behalf!
You may have already framed your next question. Why bother to redirect the
standard input from a file if the command can read the file itself? After all, wc can also
use a filename as argument! The answer is that there are times when you need to keep
the command ignorant of the source of its input. This aspect, representing one of the
most deep-seated features of the system, will gradually expose itself as you progress
through this text.

Note

When the standard input is redirected to come from a file (with <), it’s the shell that opens the
file. The command here is totally ignorant of what the shell is doing to provide it with input.
However, when you invoke a command with a filename as argument, it’s the command that
opens the file and not the shell.

Taking Input Both from File and Standard Input When a command takes input
from multiple sources—say, a file and standard input—the - symbol must be used to
indicate the sequence of taking input. The meaning of the following sequences should
be quite obvious:
cat - foo
cat foo - bar

First from standard input and then from foo
First from foo, then standard input, and then bar

Use /dev/stdin in case the - doesn’t work. The third source of standard input is the
pipe, which is discussed later (6.8). There’s a fourth form of standard input that we have

das76205_Ch06_156-187.indd 167

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

168

not mentioned here. It’s the here document, which has application in shell programming
and hence is discussed in Chapter 13.
6.5.2

Standard Output
All commands displaying output on the terminal actually write to the standard output
file as a stream of characters, and not directly to the terminal as such. There are three
possible destinations for this stream (Fig. 6.2):
• The terminal, the default destination.
• A file, using the redirection symbols > and >>.
• As input to another program using a pipeline (to be taken up later).
The shell can effect redirection of this stream when it sees the > or >> symbols in the
command line. You can replace the default destination (the terminal) with any file by
using the > (right chevron) operator, followed by the filename:
$ wc /etc/passwd > newfile
$ cat newfile
21
45
1083 /etc/passwd

The first command sends the word count of /etc/passwd to newfile; nothing appears
on the terminal screen. The sequence works like this:
1. On seeing the >, the shell opens the disk file, newfile, for writing.
2. It unplugs the standard output file from its default destination and assigns it to
newfile.
3. wc (and not the shell) opens the file /etc/passwd for reading.
4. wc writes to standard output, which was previously reassigned by the shell to
newfile.
And all of this happens without wc knowing that it is in fact writing to newfile! Any
command that uses standard output is also ignorant of the destination of its output.
Consider this sequence that uses cat to create a file:
FIGURE 6.2

The Three Destinations of Standard Output
Pipe
Terminal
(Default)

command
The quick brown fox jumbed
over the lazy dog. The quick
brown fox jumbed over the
lazy dog. The quick brown fox
jumbed over the lazy dog. The
quick brown fox jumbed over
the lazy dog.

das76205_Ch06_156-187.indd 168

File

12/13/11 10:44 AM

Chapter 6: The Shell

169

$ cat > foo
cat without an argument takes input from standard input.
In this case, the keyboard input makes its way to redirected standard output.
[Ctrl-d]

When you enter cat > foo, the shell associates cat’s standard output with the file foo.
Because the command was used without an input filename, cat looks for input from the
standard input. Enter these two lines as shown above and terminate standard input with
[Ctrl-d]. The file foo, which was created by the shell, now contains these two lines.
If the output file doesn’t exist, the shell creates it before executing the command. If
it exists, the shell overwrites it, so use this operator with caution. The shell also provides
the >> symbol (the right chevron used twice) to append to a file:
wc sample.txt >>newfile

Doesn’t disturb existing contents

Redirection can also be used with multiple files. The following example saves all C
programs:
cat *.c > c_progs_all.txt

The standard output of one command can also be used by another command as its
standard input. This is the third destination of standard output and is taken up in the
discussion on pipes (6.8).

Note

6.5.3

When the output of a command is redirected to a file, the output file is created or truncated
by the shell before the command is executed. Any idea what cat foo > foo does?

The File Descriptor
Before we proceed any further, you should know that each of the three standard files
is represented by a number, called a file descriptor. A file is opened by referring to its
pathname, but subsequent read and write operations identify the file by this file descriptor.
The kernel maintains a table of file descriptors for every process running in the system.
The first three slots are generally allocated to the three standard streams in this manner:
0—Standard input
1—Standard output
2—Standard error
These descriptors are implicitly prefixed to the redirection symbols. For instance, >
and 1> mean the same thing to the shell, while < and 0< also are identical. We need to
explicitly use one of these descriptors when handling the standard error stream. If your
program opens a file, in all probability, the file will be allocated the descriptor 3.

das76205_Ch06_156-187.indd 169

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

170

How Redirection Works
The concepts related to redirection are pretty simple. A command like ls writes to
file descriptor 1, and this remains true even when you use ls > foo. To save the ls
output in foo, the shell has to manipulate this file descriptor before running ls. It
closes the standard output and then opens foo. Since the kernel allocates the lowest
unallocated integer in the file descriptor table, foo is assigned the value 1. The ls
output is thus captured in foo.
Even though the concept appears simple, its implementation requires two processes.
After all, if the shell closes its own standard output file, how does it display its own
messages? In reality, the shell creates a copy of its own process, performs the descriptor manipulation in the copied process, and even runs the ls command in that
process. The shell’s own file descriptors are then left undisturbed. Chapter 18 discusses how the dup, dup2, and fcntl system calls are used to implement redirection.

6.5.4

Standard Error
When a command runs unsuccessfully, diagnostic messages often show up on the screen.
This is the standard error stream whose default destination is the terminal. Trying to
“cat” a nonexistent file produces the error stream:
$ cat foo
cat: cannot open foo

cat fails to open the file and writes to the standard error. If you are not using the C shell,
you can redirect this stream to a file. Using the symbol for standard output obviously
won’t do; you need to use the 2> symbols:
$ cat foo > errorfile
cat: cannot open foo
$ cat foo 2> errorfile
$ cat errorfile
cat: cannot open foo

Error stream can’t be captured with >

Even though standard output and standard error use the terminal as the default destination,
the shell possesses a mechanism for capturing them individually. You can also append
standard error in the same way you append standard output:
cat foo 2>> errorfile

or redirect them separately:
foo.sh > bar1 2>bar2

What about saving both streams in the same file? The descriptor replicating mechanism,
which does this job, is taken up shortly.

das76205_Ch06_156-187.indd 170

12/13/11 10:44 AM

171

Chapter 6: The Shell

C Shell

Tip

6.5.5

The standard error is handled differently by the C shell, so the examples of this section
won’t work with it. In fact, the C shell merges the standard error with the standard
output; it has no separate symbol for handling standard error only. The command
cat foo >& bar saves both standard output and standard error in bar.

If you have a program that runs for a long time and is not error-free, you can direct the standard error to a separate file and then stay away from the terminal. On return, you can examine
this file for errors.

Filters—Using Both Standard Input and Standard Output
We return to the input and output streams to ask ourselves this question: Do all commands use the features of standard input and standard output? No, certainly not. From
this viewpoint, the UNIX commands can be grouped into four categories:
Commands

Standard Input

Standard Output

mkdir, rmdir, cp, rm

No

No

ls, pwd, who

No

Yes

lp, lpr

Yes

No

cat, wc, gzip

Yes

Yes

Commands in the fourth category are called filters, and the dual stream-handling feature
makes filters powerful text manipulators. Note that most filters can also read directly from
files whose names are provided as arguments. Chapters 9, 10, 12, and 14 are earmarked
for filters in this text. Some of the commands discussed in previous chapters can also
be made to behave as filters.
Since wc is a filter, you can redirect wc’s standard input to come from a file and
save the output in yet another. This can be done in any of these ways:
wc < calc.txt > result.txt
wc > result.txt < calc.txt
wc>result.txt result.txt < calc.txt wc

Using both standard input and output
No whitespace!
As above, but command at end

The last example illustrates a departure from the statement made previously (2.2) that
the first word in the command line is the command. In the last example, wc is the last
word in the command line. Rather than use these combinations, you’ll find it more
convenient to stick to the first form.
The indifference of a command to the source of its input and destination of its
output is one of the most profound features of the UNIX system. It raises the possibility of commands “talking” to one another so that the output of one command can be
used as input to another. Very soon we’ll discuss pipes, and you’ll see how two or more
commands communicate with one another.

das76205_Ch06_156-187.indd 171

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

172

6.6 Collective Manipulation
So far, we have used the > to handle a single stream of a single command. But the shell
also supports collective stream handling. This can happen in these two ways:
• Handle two standard streams as a single one using the 2>&1 and 1>&2 symbols.
• Form a command group by enclosing multiple commands with the ( and ) symbols
or { and } symbols. You can then use a single instruction to control all commands
in the group.
Both concepts have important applications. We must understand them now because these
features will be recalled when discussing processes, shell programming, and systems
programming.
6.6.1

Replicating Descriptors
Though standard output and standard error are two separate streams, you can manipulate
them collectively. It’s like redirecting a single stream. The symbols required are a little
cryptic to look at but quite intuitive:
1>&2
2>&1

Send the standard output to the destination of the standard error.
Send the standard error to the destination of the standard output.

In either case, both streams are associated with a single file. The first set is often used
inside shell scripts in this way:
echo “Filename not entered” 1>&2

Same as >&2

Note that the echo statement has not been redirected here. If a script containing this statement is redirected, then the output of echo won’t go the file but will be sent to the standard
error, the terminal. To save this message in a file, you have to provide redirection separately:
$ echo “Filename not entered” 2>error.txt 1>&2
$ cat errror.txt
Filename not entered

The 2> symbol reassigns standard error to error.txt and 1>&2 sends the standard output
of echo to the standard error. Note the sequence: first we redirect and then we specify
the replication of the descriptor.

Tip

6.6.2

das76205_Ch06_156-187.indd 172

Some programs (like perldoc) are designed to write to the standard error. Piping the output to
more doesn’t help. To use the pager with these programs also, use 2>&1 to send standard error
to the standard output. If you run perldoc perlop 2>&1 | more, you’ll be able to separately
view each page of the perl documentation. However, this may not work on all systems.

Command Grouping
Sometimes, we need to manipulate a group of commands collectively: redirect them,
run them in the background, and so on. The () and {} handle a command group.

12/13/11 10:44 AM

Chapter 6: The Shell

173

We can use the first set to redirect the standard output of a command group using a
single >:
( ls -x *.c ; echo ; cat *.c ) > c_progs_all.txt

This saves all C program sources in a file preceded by a multicolumn list of programs
acting as a table of contents. The echo command serves to insert a blank line between
them. The {} can also be used for this purpose:
{ ls -x *.c ; echo ; cat *.c ; } > c_progs_all.txt

Note the ; at the end of the cat command that is required if the opening and closing curly
braces are placed in the same line. You don’t need it when the } is located in a separate line.
Though we could use the two sets of symbols interchangeably here, there are
distinct differences between them. After we study processes, we’ll be able to identify
those situations where one group applies and not the other. We’ll use the curly braces
extensively when programming with the shell.

6.7 /dev/null and /dev/tty: Two Special Files
/dev/null Quite often, and especially in shell programming, you’ll want to check
whether a program runs successfully without seeing its output or saving it in a file. You
have a special file that accepts any stream without growing in size—the file /dev/null:
$ cat /etc/passwd >/dev/null
$ cat /dev/null
$ _

Size is always zero

Check the file size; it’s always zero. /dev/null simply incinerates all output written to
it. This facility is also useful in redirecting error messages. Consider the find command
that was used in the Tip in Section 4.11.1:
find / -name typescript -print 2>/dev/null

The file /dev/null is actually a pseudo-device because, unlike all other device files,
it’s not associated with any physical device.
The file /dev/null can also be used as a “dummy” file when we need a command to work
with two files but only one is available. Consider the grep command (10.2), which displays lines
containing a pattern. The command also displays filenames when used with multiple filenames
as arguments. But since the command used with the -exec operator of find (4.11.3) works
with a single file at a time, we can use /dev/null to provide an additional argument to grep:
Tip

find . -name “*.c” -exec grep printf {} /dev/null \;
This command now shows the filename prepended to every line containing printf. Without
/dev/null, grep would output only the line containing the pattern, and you wouldn’t know
which file the line comes from.

das76205_Ch06_156-187.indd 173

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

174

/dev/tty The second special file in the UNIX system is the one indicating one’s
terminal—/dev/tty. But make no mistake: This is not the file that represents standard
output or standard error. Commands generally don’t write to this file, but you’ll be
required to redirect some statements in shell scripts to this file.
Consider, for instance, that romeo is working on terminal /dev/pts/1 and juliet
on /dev/pts/2. Both romeo and juliet, however, can refer to their own terminals with
the same filename—/dev/tty. Thus, if romeo issues the command
who >/dev/tty

the list of current users is sent to the terminal he is currently using—/dev/pts/1.
Similarly, juliet can use an identical command to see the output on her terminal,
/dev/pts/2. Like /dev/null, /dev/tty can be accessed by several users without conflict.
You may ask why one would need to specifically redirect output to one’s own
terminal since the default output goes to the terminal anyway. Sometimes, you need to
specify that explicitly. Apart from its use in redirection, /dev/tty can also be used as
an argument to some UNIX commands. Section 6.9 makes use of this feature, while
some situations are presented in Chapter 13 (featuring shell programming).
The size of /dev/null is always zero, and all terminals can be represented by /dev/tty.
Note

6.8 Pipes
To understand pipes, we’ll set ourselves the task of counting the number of users currently
logged in. We’ll first attempt the task using the knowledge we possess already. The who
command produces a list of users—one user per line, and we’ll save this output in a file:
$ who > user.txt
$ cat user.txt
root
console
romeo
pts/10
juliet
pts/6
project pts/8

Aug
Aug
Aug
Aug

1
1
1
1

07:51
07:56
02:10
02:16

(:0)
(pc123.heavens.com)
(pc125.heavens.com)
(pc125.heavens.com)

If we now redirect the standard input of the wc -l command (3.18) to come from
user.lst, we will have effectively counted the number of users:
$ wc -l < user.txt
4

Counts the number of users

This method of running two commands separately has two obvious disadvantages:
• For long-running commands, this process can be slow. The second command can’t
act unless the first has completed its job.
• You require an intermediate file that has to be removed after completion of the job.
When you are handling large files, temporary files can build up easily and eat up
disk space in no time.

das76205_Ch06_156-187.indd 174

12/13/11 10:44 AM

Chapter 6: The Shell

175

Here, who’s standard output was redirected, as was wc’s standard input, and both used
the same disk file. The shell can connect these streams using a special operator—the |
(pipe)—and avoid the creation of the disk file. You can make who and wc work in tandem
so that one takes input from the other:
$ who | wc -l
4

No intermediate files created

The output of who has been passed directly to the input of wc, and who is said to be piped
to wc. When a sequence of commands is combined together in this way, a pipeline is
formed. The shell sets up this interconnection; the commands have no knowledge of it.
The pipe is the third source and destination of standard input and standard output,
respectively. You can now use one to count the number of files in the current directory:
$ ls | wc -l
15

Note that no separate command was designed to count files, though the designers could
easily have provided another option to ls to perform this operation. And because wc
uses standard output, you can redirect this output to a file:
ls | wc -l > fkount

You can run a large number of commands in a pipeline, the number being restricted to
the maximum number of processes that can be run by a non-root user. But you must
know the behavioral properties of these commands to place them there. Consider this
command sequence, which prints the man page of grep on the printer:
man grep | col -b | lp

Don’t print these pages!

The online man pages of a command often show the keywords in boldface. These
pages contain a number of control characters which are removed here by the col -b
command. Like col, lp also reads its standard input from col’s output and prints the
file. For a pipeline like this to be feasible, the leftmost command (here, man) must be
able to write to standard output while the rightmost command (here, lp) must be able to
read from standard input. Intermediate commands (here, col) must be able to do both,
i.e., behave like a filter.
6.8.1

When a Command Needs to Be Ignorant of Its Source
We’ve made several references to a command being ignorant of its source and destination. When and why is this ignorance essential for us? To appreciate this point, let’s use
the wc command to display the total size of all C programs:
$ wc -c *.c
2078 backtalk.c
231 beyond_array.c
.......
1944 dog.c

das76205_Ch06_156-187.indd 175

Output trimmed here

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

176
884 hexdump.c
214 swap.c
940101 total

The display shows the total usage at 940,101 bytes, but it also shows the usage for each
file. We are not interested in individual statistics this time; what we need is a single
figure representing the total size. To do that, you must make wc ignorant of its input
source. You can accomplish this by feeding the concatenated output stream of all these
files to wc -c as its input:
$ cat *.c | wc -c
940101

When do we need a single figure? We can use this command sequence as a control command in a shell script to determine whether the files will fit on a diskette:
if [ `cat *.c | wc -c` -lt 1474560 ] ; then
echo ‘These files will fit in a single 3.5” diskette’
fi

Note how the single quotes protect the double quote in the echo statement. We’ll learn
to use the if construct in Chapter 13, which features shell programming. The two backquotes (``) denote command substitution, which is discussed shortly.
In a pipeline, the command on the left of the | must use standard output, and the one on the
right must use standard input.

Note

6.9 tee: Creating a Tee
tee is an external command and not a feature of the shell. It duplicates its input, saves
one copy in a file, and sends the other to the standard output. Since it is also a filter,
tee can be placed anywhere in a pipeline. The following command sequence uses tee
to display the output of who and save the output in a file as well:
$ who | tee user.txt
romeo
pts/2
juliet
pts/3
sumit
pts/5

Sep 7 08:41
Sep 7 17:58
Sep 7 18:01

(pc123.heavens.com)
(pc122.heavens.com)
(mercury.heavens.com)

tee doesn’t perform any filtering action on its input; it gives out exactly what it takes.
You can crosscheck the display with the contents of the file user.txt:
$ cat user.txt
romeo
pts/2
juliet
pts/3
sumit
pts/5

das76205_Ch06_156-187.indd 176

Sep 7 08:41
Sep 7 17:58
Sep 7 18:01

(pc123.heavens.com)
(pc122.heavens.com)
(mercury.heavens.com)

12/13/11 10:44 AM

177

Chapter 6: The Shell
You can pipe tee’s output to another command, say wc:
$ who | tee user.txt | wc -l
3

How do you use tee to display both the list of users and its count on the terminal? Since
the terminal is also a file, you can use the device name /dev/tty as an argument to tee:
$ who | tee /dev/tty | wc -l
romeo
pts/2
Sep 7 08:41
juliet
pts/3
Sep 7 17:58
sumit
pts/5
Sep 7 18:01
3

/dev/tty used as command argument

(pc123.heavens.com)
(pc122.heavens.com)
(mercury.heavens.com)

The advantage of treating the terminal as a file is apparent from the preceding example.
You couldn’t have done so if tee (or, for that matter, any UNIX command) had placed
restrictions on the type of file it could handle. Here the terminal is treated in the same
way as any disk file.

6.10 Command Substitution
The shell enables the connection of two commands in yet another way. While a pipe
enables a command to obtain its standard input from the standard output of another
command, the shell enables one or more command arguments to be obtained from the
standard output of another command. This feature is called command substitution.
To consider a simple example, suppose you need to display today’s date with a
statement like this:
The date today is Sat Sep 7 19:01:16 GMT 2002

The last part of the statement (beginning from Sat) represents the output of the date
command. How does one incorporate date’s output into the echo statement? With command substitution it’s a simple matter. Use the expression `date` as an argument to echo:
$ echo The date today is `date`
The date today is Sat Sep 7 19:01:56 GMT 2002

When scanning the command line, the ` (backquote or backtick) is another metacharacter
that the shell looks for. There’s a special key on your keyboard (generally at the top-left)
that generates this character, and it should not be confused with the single quote (‘).
The shell executes the enclosed command and replaces the enclosed command line with
the output of the command. For command substitution to work, the command so “backquoted” must use standard output. date does; that’s why command substitution worked.
Commands that use filenames as arguments can use command substitution to
obtain their arguments from a list:
ls `cat filelist`

das76205_Ch06_156-187.indd 177

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

178

Here, filelist contains a list of filenames. You can also use this feature to generate
useful messages. For example, you can use two commands in a pipeline and then use
the output as the argument to a third:
$ echo “There are `ls | wc -l` files in the current directory”
There are 58 files in the current directory

The command worked properly even though the arguments were double-quoted. It’s a
different story altogether when single quotes are used:
$ echo ‘There are `ls | wc -l` files in the current directory’
There are `ls | wc -l` files in the current directory

This was to be expected because we had already tried out a similar exercise earlier (6.4.2).
The ` is one of the few characters interpreted by the shell when placed within double
quotes. If you want to echo a literal `, you have to use single quotes.
Command substitution has interesting application possibilities in shell scripts. It
speeds up work by letting you combine a number of instructions in one. You’ll see more
of this feature in subsequent chapters.

Note

KORN Shell

BASH Shell

Command substitution is enabled when backquotes are used within double quotes. If you use
single quotes, it’s not.

POSIX recommends the use of the form $(command) rather than the archaic `command`
for command substitution. The Korn and Bash shells offer both forms. The POSIX form
requires you to place the command inside parentheses and have a $ before them:
$ echo The date today is $(date)
The date today is Sat Sep 7 19:15:33 GMT 2002

Whether or not you should use the POSIX notation is something you have to decide
for yourself. Make sure that you don’t have to run your shell scripts with the Bourne
shell before you decide to adopt this form.

6.11 Shell Variables
The shell supports variables that are useful both in the command line and shell scripts.
You have already encountered some of them like HOME and SHELL. Variable usage in
the Bourne family differs from that in the C shell. In this section and elsewhere, we
discuss Bourne-type variables. The features of C shell variables are noted in the aside
at the end of this section.
A variable assignment is of the form variable=value (no spaces around =), but its
evaluation requires the $ as prefix to the variable name:
$ count=5
$ echo $count
5

das76205_Ch06_156-187.indd 178

No $ required for assignment
but needed for evaluation

12/13/11 10:44 AM

179

Chapter 6: The Shell
A variable can also be assigned the value of another variable:
$ total=$count
$ echo $total
5

Caution

Assigning a value to another variable

Programmers should note that there must not be any whitespace on either side of the =
symbol. The command line x =5 is interpreted by the shell as the x command running with
the =5 argument!

Variable concatenation is simple; no special operators or symbols are needed. Simply
place the variables side by side:
$ ext=.avi
$ moviename=holmes
$ filename=$moviename$ext
$ echo $filename
holmes.avi

Variable names begin with a letter but can contain numerals and the _ as the other characters. Names are case-sensitive; x and X are two different variables. Unlike in programming languages, shell variables are not typed; you don’t need to use a char, int, or long
prefix when you define them. In fact, you don’t even have to declare them before you can
use them. All shell variables are of the string type, which means that even a number like
123 is stored as a string rather than in binary. (This may not remain true in the future.)
All shell variables are initialized to null strings by default. While explicit assignment of null strings with x=”” or x=’’ is possible, you can also use this as a shorthand:
x=

A null string

A variable can be removed with unset and protected from reassignment by readonly.
Both are shell internal commands:
unset x
readonly x

Tip

x is now undefined
x can’t be reassigned

By convention, variable names used by the UNIX system and software packages are in uppercase. You are advised to use lowercase variable names in your shell scripts simply to distinguish
them from system variables.

The C shell uses the set statement to set variables. There either has to be whitespace
on both sides of the = or none at all:
C Shell

set count = 1
set count=1

Both statements are valid
but set count= 1 won’t work

The evaluation is done in the normal manner (echo $count). The C shell uses another
statement, setenv, to set an environment variable. These variables are discussed in
Chapter 8.

das76205_Ch06_156-187.indd 179

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

180
6.11.1

Effects of Quoting and Escaping
To assign a multiword string to a variable, you can escape each space character, but
quoting (single or double) is the preferred solution:
message=You\ didn’t\ enter\ the\ filename
message=”You didn’t enter the filename”

Now that you have another special character ($) that is gobbled up by the shell, you may
still need to interpret it literally without it being evaluated. This time we have escaping
and single-quoting as our options:
$ echo The average pay is \$1000
The average pay is $1000
$ echo ‘The average pay is $1000’
The average pay is $1000

Like the backquote, the $ is also evaluated by the shell when it is double-quoted. Here
are two examples:
$ echo “The
The PATH is
$ echo “The
The average

PATH is $PATH and the current directory is `pwd`”
/bin:/usr/bin:. and the current directory is /home/romeo/workc
average pay is $1000”
pay is 000

The first example shows both command substitution and variable evaluation at work;
but have a look at the second example. Here, the shell evaluated a “variable” $1. It is
not defined, so a null string was output. $1 belongs to a set of parameters that are called
positional parameters (13.3), signifying the arguments that you pass to a script.

Note

6.11.2

Whether you use double or single quotes depends on whether you want command substitution and variable evaluation to be enabled or not. Double quotes permit their interpretation,
but single quotes don’t.

Where to Use Shell Variables
Setting Pathnames If a pathname is used several times in a script, you should assign
it to a variable. You can then use it as an argument to any command. Let’s use it with
cd in this manner:
$ progs=’/home/romeo/c_progs’
$ cd $progs ; pwd
/home/romeo/c_progs

A shell script would generally contain this definition at the beginning, and then it could
be used everywhere—both in the script and in other scripts run from that script. This
assignment means less typing, but there’s another advantage. In a later reorganization,
if the location of c_progs changes to, say, /export/home/romeo/c_progs, then you

das76205_Ch06_156-187.indd 180

12/13/11 10:44 AM

Chapter 6: The Shell

181

simply need to change the variable definition, and everything will work in the same
way as before.
Using Command Substitution You can also use the feature of command substitution to set variables:
$ mydir=`pwd` ; echo $mydir
/home/romeo/c_progs

You can store the size of a file in a variable too:
size=`wc -c < foo.txt`

We used the < symbol to leave out the filename in the value assigned to size. If we had
used foo.txt as an argument instead, then size would have contained a two-word string.
The UNIX system also uses a number of variables to control its behavior. There
are variables that tell you the type of terminal you are using, the prompt string that
you use, or the directory where incoming mail is kept. These variables are often called
environment variables because they are available in all processes owned by a user. (The
variable mydir or size is not.) A detailed discussion on the significance of the major
environment variables will be taken up in Chapter 8.

6.12 Shell Scripts
The shell offers the facility of storing a group of commands in a file and then executing
the file. All such files are called shell scripts. You’ll also find people referring to them
as shell programs and shell procedures. The instructions stored in these files are executed
in the interpretive mode—much like the batch (.BAT) files of Windows.
The following shell script has a sequence of three commands stored in a file
script.sh. You can create the file with vi or pico, but since this takes only three lines,
you can use cat instead:
$ cat > script.sh
directory=`pwd`
echo The date today is `date`
echo The current directory is $directory
[Ctrl-d]
$ _

Beginning of standard input

End of standard input

The extension .sh is used only for the purpose of identification; it can have any extension or even none. Try executing the file containing these commands by simply invoking
the filename:
$ script.sh
script.sh: execute permission denied

das76205_Ch06_156-187.indd 181

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

182

Executable permission is usually necessary for any shell script to run, and by default, a
file doesn’t have this permission on creation. Use chmod to first accord executable status
to the file before executing it:
$ chmod u+x script.sh
$ script.sh
The date today is Thu Feb 17 11:30:53 EST 2000
The current directory is /home/romeo/project5

The script executes the three statements in sequence. Even though we used the shell as
an interpreter, it is also a programming language. You can have all the standard constructs like if, while, and for in a shell script. The behavior of the UNIX system is
controlled by many preinstalled shell scripts that are executed during system startup and
those written by the system administrator. We explore shell programming in Chapter 13.

6.13

The Shell’s Treatment of the Command Line
Now that you have seen the major interpretive features of the shell, it’s time we made a
summary of these activities. After the command line is terminated by hitting the [Enter]
key, the shell goes ahead with processing the command line in one or more passes. The
sequence varies with the shell you use, but broadly assumes the following order:
• Parsing The shell first breaks up the command line into words using spaces and
tabs as delimiters, unless quoted. All consecutive occurrences of a space or tab are
replaced here with a single space.
• Variable evaluation All words preceded by a $ are evaluated as variables, unless
quoted or escaped.
• Command substitution Any command surrounded by backquotes is executed
by the shell, which then replaces the standard output of the command in the command line.
• Redirection The shell then looks for the characters >, <, and >> to open the files
they point to.
• Wild-card interpretation The shell finally scans the command line for wild cards
(the characters *, ?, [ and ]). Any word containing a wild card is replaced with
a sorted list of filenames that match the pattern. The list of these filenames then
forms the arguments to the command.
• PATH evaluation The shell finally looks for the PATH variable to determine the
sequence of directories it has to search in order to hunt for the command.
The preceding sequence can be considered a simplistic treatment of the shell’s behavioral
pattern. There are many more characters that the shell looks for that have been ignored
here. And the shell itself can be viewed from different perspectives. Chapter 7 examines
the process of the shell. Later chapters describe the shell’s environment (Chapter 8) and
its programming features (Chapter 13).

das76205_Ch06_156-187.indd 182

12/13/11 10:44 AM

183

Chapter 6: The Shell
➤

6.14

GOING FURTHER

More Wild Cards
Matching Totally Dissimilar Patterns This feature, not available in the Bourne
shell, enables us to match totally dissimilar patterns. How does one copy all the C and
Java source programs from another directory? Delimit the patterns with a comma, and
then put curly braces around them (no spaces please!):
cp $HOME/prog_sources/*.{c,java} .

Won’t work in Bourne shell

This works in the Korn, Bash and C shells. The Bourne shell would require two separate invocations of cp to do this job. Using the curly brace form, you can also access
multiple directories:
cp /home/romeo/{project,html,scripts}/* .

Won’t work in Bourne shell

This copies all files from three directories (project, html, and scripts) to the current
directory. Isn’t this convenient?

!(*.exe)

All files without .exe extension

GOING FURTHER

The Invert Selection Feature If you have used Windows Explorer, you would no
doubt have used the Invert Selection feature. This option reverses the selection you make
with your mouse and highlights the rest. Bash and Korn also provide a similar feature
of matching all filenames except those in the expression. For instance, this expression

matches all except the .exe files. If you want to include multiple expressions in the
exception list, then use the | as the delimiter:
cp !(*.jpg|*.jpeg|*.gif) ../text

This copies all except the graphic files in GIF or JPEG format to the text directory. Note
that the parentheses and | can be used to group filenames only if the ! precedes the group.

BASH Shell

The exclusion feature won’t work in Bash unless you make the setting
shopt -s extglob. Even if you don’t understand what this means, simply place
this statement in .bash_profile or .profile, whichever is your startup file (8.9.1).

6.15 xargs: Building a Dynamic Command Line
Sometimes, the filenames used by commands can be determined only at runtime. UNIX
provides a real dark horse—the xargs command—that can run any command but obtains
the file list from standard input. This feature is often used to handle the problem created by find’s -exec operator. If find produces a list of 200 files for removal with
-exec rm {} \;, the rm command has to be executed 200 times.

das76205_Ch06_156-187.indd 183

12/13/11 10:44 AM

Your UNIX/Linux: The Ultimate Guide

184

xargs comes to our rescue here as it lets rm (or, for that matter, any UNIX command) be used just once with 200 filenames as arguments. The following command lines
do the same thing except that the second one does it much faster:
find /usr/preserve -mtime +30 -exec rm -f {} \;
find /usr/preserve -mtime +30 -print | xargs rm -f

xargs here obtains the file list from find and supplies a single set of arguments to rm.
So even if find selects 30 files, rm is executed only once. You could say that command
substitution can also do the same job, but xargs has other advantages.
Commands usually have limits on the number of arguments they can handle. xargs
uses the -n option to provide the specified number of arguments for a single invocation
of the command:
find / -name core -size +1024 -print | xargs -n20 rm -f

GOING FURTHER

If find locates 100 files, rm will be invoked five times—each time with 20 filenames
as arguments. A useful tool indeed!

das76205_Ch06_156-187.indd 184

SUMMARY
The shell is a program that runs when a user logs in and terminates when she logs out.
It scans the command line for metacharacters and rebuilds it before turning it over to
the kernel for execution. The shell may or may not wait for the command to terminate.
The shell matches filenames with wild cards. It can match any number of any
characters (*) or a single one (?). It can also match a character class ([ ]) and negate
a match ([!]). The * doesn’t match a filename beginning with a dot.
A wild card is escaped with a \ to be treated literally, and if there are a number
of them, then they should be quoted. Single quotes protect all special characters, while
double quotes enable command substitution and variable evaluation.
Files are accessed with small integers called file descriptors. The shell makes
available three files representing standard input, standard output, and standard error to
every command that it runs. It manipulates the default source and destination of these
streams by assigning them to disk files.
The file /dev/null never grows in size, and every user can access her own terminal as /dev/tty.
Pipes connect the standard output of one command to the standard input of another.
Commands using standard output and standard input are called filters. A combination
of filters placed in pipelines can be used to perform complex tasks that the commands
can’t perform individually.
The external tee command duplicates its input. It saves one to a file and writes
the other to the standard output.
Command substitution enables a command’s standard output to become the arguments of another command.
The shell supports variables, which are evaluated by prefixing a $ to the variable name. The variables that control the workings of the UNIX system are known as
environment variables.

12/13/11 10:44 AM

185

Chapter 6: The Shell

The shell is also a scripting language, and a group of commands can be placed in
a shell script to be run in a batch.

SELF-TEST
6.1
6.2
6.3
6.4
6.5
6.6

6.7
6.8
6.9
6.10
6.11
6.12
6.13
6.14
6.15
6.16
6.17
6.18
6.19
6.20
6.21
6.22

das76205_Ch06_156-187.indd 185

Why does the shell need to expand wild cards? How does it treat the * when
used as an argument to a command (like echo *)?
What is the significance of the command ls *.*? Does it match filenames that
begin with a dot?
How do you remove only the hidden files of your directory? Does rm * remove
these files as well?
Match the filenames chapa, chapb, chapc, chapx, chapy, and chapz with a
wild-card expression.
Is the wild-card expression [3-h]* valid?
Devise a command that copies all files named chap01, chap02, chap03, and
so forth through chap26 to the parent directory. Can a single wild-card pattern
match them all?
Frame wild-card patterns (i) where the last character is not numeric, (ii) that have
at least four characters.
When will cd * work?
Which UNIX command uses wild cards as part of its syntax?
How do you split a long command sequence into multiple lines?
Name the three sources and destinations of standard input and standard output.
Is the output of the command cat foo1 foo2 >/dev/tty directed to the standard output?
Is this a legitimate command, and what does it appear to do? >foo