The Definitive Guide

• Table of Contents
• Preface
  • Running Example: Joe’s Hardware Store
  • Chapter-by-Chapter Guide
  • Typographic Conventions
  • Comments and Questions
  • Acknowledgments
• Part I
• Overview of HTTP
  • HTTP: The Internet’s Multimedia Courier
  • Web Clients and Servers
  • Resources
    • Media Types
    • URIs
    • URLs
    • URNs
  • Transactions
    • Methods
    • Status Codes
    • Web Pages Can Consist of Multiple Objects
  • Messages
    • Simple Message Example
  • Connections
    • TCP/IP
    • Connections, IP Addresses, and Port Numbers
    • A Real Example Using Telnet
  • Protocol Versions
  • Architectural Components of the Web
    • Proxies
    • Caches
    • Gateways
    • Tunnels
    • Agents
  • The End of the Beginning
  • For More Information
    • HTTP Protocol Information
    • Historical Perspective
    • Other World Wide Web Information
• URLs and Resources
  • Navigating the Internet’s Resources
    • The Dark Days Before URLs
  • URL Syntax
    • Schemes: What Protocol to Use
    • Hosts and Ports
    • Usernames and Passwords
    • Paths
    • Parameters
    • Query Strings
    • Fragments
  • URL Shortcuts
    • Relative URLs
      • Base URLs
      • Resolving relative references
    • Expandomatic URLs
  • Shady Characters
    • The URL Character Set
    • Encoding Mechanisms
    • Character Restrictions
    • A Bit More
  • A Sea of Schemes
  • The Future
    • If Not Now, When?
  • For More Information
• HTTP Messages
  • The Flow of Messages
    • Messages Commute Inbound to the Origin Server
    • Messages Flow Downstream
  • The Parts of a Message
    • Message Syntax
    • Start Lines
      • Request line
      • Response line
      • Methods
      • Status codes
      • Reason phrases
      • Version numbers
    • Headers
      • Header classifications
      • Header continuation lines
    • Entity Bodies
    • Version 0.9 Messages
  • Methods
    • Safe Methods
    • GET
    • HEAD
    • PUT
    • POST
    • TRACE
    • OPTIONS
    • DELETE
    • Extension Methods
  • Status Codes
    • 100–199: Informational Status Codes
      • Clients and 100 Continue
      • Servers and 100 Continue
      • Proxies and 100 Continue
    • 200–299: Success Status Codes
    • 300–399: Redirection Status Codes
    • 400–499: Client Error Status Codes
    • 500–599: Server Error Status Codes
  • Headers
    • General Headers
      • General caching headers
    • Request Headers
      • Accept headers
      • Conditional request headers
      • Request security headers
      • Proxy request headers
    • Response Headers
      • Negotiation headers
      • Response security headers
    • Entity Headers
      • Content headers
      • Entity caching headers
  • For More Information
• Connection Management
  • TCP Connections
    • TCP Reliable Data Pipes
    • TCP Streams Are Segmented and Shipped by IP Packets
    • Keeping TCP Connections Straight
    • Programming with TCP Sockets
  • TCP Performance Considerations
    • HTTP Transaction Delays
    • Performance Focus Areas
    • TCP Connection Handshake Delays
    • Delayed Acknowledgments
    • TCP Slow Start
    • Nagle’s Algorithm and TCP_NODELAY
    • TIME_WAIT Accumulation and Port Exhaustion
  • HTTP Connection Handling
    • The Oft-Misunderstood Connection Header
    • Serial Transaction Delays
  • Parallel Connections
    • Parallel Connections May Make Pages Load Faster
    • Parallel Connections Are Not Always Faster
    • Parallel Connections May “Feel” Faster
  • Persistent Connections
    • Persistent Versus Parallel Connections
    • HTTP/1.0+ Keep-Alive Connections
    • Keep-Alive Operation
    • Keep-Alive Options
    • Keep-Alive Connection Restrictions and Rules
    • Keep-Alive and Dumb Proxies
      • The Connection header and blind relays
      • Proxies and hop-by-hop headers
    • The Proxy-Connection Hack
    • HTTP/1.1 Persistent Connections
    • Persistent Connection Restrictions and Rules
  • Pipelined Connections
  • The Mysteries of Connection Close
    • “At Will” Disconnection
    • Content-Length and Truncation
    • Connection Close Tolerance, Retries, and Idempotency
    • Graceful Connection Close
      • Full and half closes
      • TCP close and reset errors
      • Graceful close
  • For More Information
    • HTTP Connections
    • HTTP Performance Issues
    • TCP/IP
• Part II
• Web Servers
  • Web Servers Come in All Shapes and Sizes
    • Web Server Implementations
    • General-Purpose Software Web Servers
    • Web Server Appliances
    • Embedded Web Servers
  • A Minimal Perl Web Server
  • What Real Web Servers Do
  • Step 1: Accepting Client Connections
    • Handling New Connections
    • Client Hostname Identification
    • Determining the Client User Through ident
  • Step 2: Receiving Request Messages
    • Internal Representations of Messages
    • Connection Input/Output Processing Architectures
  • Step 3: Processing Requests
  • Step 4: Mapping and Accessing Resources
    • Docroots
      • Virtually hosted docroots
      • User home directory docroots
    • Directory Listings
    • Dynamic Content Resource Mapping
    • Server-Side Includes (SSI)
    • Access Controls
  • Step 5: Building Responses
    • Response Entities
    • MIME Typing
    • Redirection
  • Step 6: Sending Responses
  • Step 7: Logging
  • For More Information
• Proxies
  • Web Intermediaries
    • Private and Shared Proxies
    • Proxies Versus Gateways
  • Why Use Proxies?
  • Where Do Proxies Go?
    • Proxy Server Deployment
    • Proxy Hierarchies
      • Proxy hierarchy content routing
    • How Proxies Get Traffic
  • Client Proxy Settings
    • Client Proxy Configuration: Manual
    • Client Proxy Configuration: PAC Files
    • Client Proxy Configuration: WPAD
  • Tricky Things About Proxy Requests
    • Proxy URIs Differ from Server URIs
    • The Same Problem with Virtual Hosting
    • Intercepting Proxies Get Partial URIs
    • Proxies Can Handle Both Proxy and Server Requests
    • In-Flight URI Modification
    • URI Client Auto-Expansion and Hostname Resolution
    • URI Resolution Without a Proxy
    • URI Resolution with an Explicit Proxy
    • URI Resolution with an Intercepting Proxy
  • Tracing Messages
    • The Via Header
      • Via syntax
      • Via request and response paths
      • Via and gateways
      • The Server and Via headers
      • Privacy and security implications of Via
    • The TRACE Method
      • Max-Forwards
  • Proxy Authentication
  • Proxy Interoperation
    • Handling Unsupported Headers and Methods
    • OPTIONS: Discovering Optional Feature Support
    • The Allow Header
  • For More Information
• Caching
  • Redundant Data Transfers
  • Bandwidth Bottlenecks
  • Flash Crowds
  • Distance Delays
  • Hits and Misses
    • Revalidations
    • Hit Rate
    • Byte Hit Rate
    • Distinguishing Hits and Misses
  • Cache Topologies
    • Private Caches
    • Public Proxy Caches
    • Proxy Cache Hierarchies
    • Cache Meshes, Content Routing, and Peering
  • Cache Processing Steps
    • Step 1: Receiving
    • Step 2: Parsing
    • Step 3: Lookup
    • Step 4: Freshness Check
    • Step 5: Response Creation
    • Step 6: Sending
    • Step 7: Logging
    • Cache Processing Flowchart
  • Keeping Copies Fresh
    • Document Expiration
    • Expiration Dates and Ages
    • Server Revalidation
    • Revalidation with Conditional Methods
    • If-Modified-Since: Date Revalidation
    • If-None-Match: Entity Tag Revalidation
    • Weak and Strong Validators
    • When to Use Entity Tags and Last-Modified Dates
  • Controlling Cachability
    • No-Cache and No-Store Response Headers
    • Max-Age Response Headers
    • Expires Response Headers
    • Must-Revalidate Response Headers
    • Heuristic Expiration
    • Client Freshness Constraints
    • Cautions
  • Setting Cache Controls
    • Controlling HTTP Headers with Apache
    • Controlling HTML Caching Through HTTP-EQUIV
  • Detailed Algorithms
    • Age and Freshness Lifetime
    • Age Computation
      • Apparent age is based on the Date header
      • Hop-by-hop age calculations
      • Compensating for network delays
    • Complete Age-Calculation Algorithm
    • Freshness Lifetime Computation
    • Complete Server-Freshness Algorithm
  • Caches and Advertising
    • The Advertiser’s Dilemma
    • The Publisher’s Response
    • Log Migration
    • Hit Metering and Usage Limiting
  • For More Information
• Integration Points: Gateways, Tunnels, and Relays
  • Gateways
    • Client-Side and Server-Side Gateways
  • Protocol Gateways
    • HTTP/*: Server-Side Web Gateways
    • HTTP/HTTPS: Server-Side Security Gateways
    • HTTPS/HTTP: Client-Side Security Accelerator Gateways
  • Resource Gateways
    • Common Gateway Interface (CGI)
    • Server Extension APIs
  • Application Interfaces and Web Services
  • Tunnels
    • Establishing HTTP Tunnels with CONNECT
      • CONNECT requests
      • CONNECT responses
    • Data Tunneling, Timing, and Connection Management
    • SSL Tunneling
    • SSL Tunneling Versus HTTP/HTTPS Gateways
    • Tunnel Authentication
    • Tunnel Security Considerations
  • Relays
  • For More Information
• Web Robots
  • Crawlers and Crawling
    • Where to Start: The “Root Set”
    • Extracting Links and Normalizing Relative Links
    • Cycle Avoidance
    • Loops and Dups
    • Trails of Breadcrumbs
    • Aliases and Robot Cycles
    • Canonicalizing URLs
    • Filesystem Link Cycles
    • Dynamic Virtual Web Spaces
    • Avoiding Loops and Dups
  • Robotic HTTP
    • Identifying Request Headers
    • Virtual Hosting
    • Conditional Requests
    • Response Handling
      • Status codes
      • Entities
    • User-Agent Targeting
  • Misbehaving Robots
  • Excluding Robots
    • The Robots Exclusion Standard
    • Web Sites and robots.txt Files
      • Fetching robots.txt
      • Response codes
    • robots.txt File Format
      • The User-Agent line
      • The Disallow and Allow lines
      • Disallow/Allow prefix matching
    • Other robots.txt Wisdom
    • Caching and Expiration of robots.txt
    • Robot Exclusion Perl Code
    • HTML Robot-Control META Tags
      • Robot META directives
      • Search engine META tags
  • Robot Etiquette
  • Search Engines
    • Think Big
    • Modern Search Engine Architecture
    • Full-Text Index
    • Posting the Query
    • Sorting and Presenting the Results
    • Spoofing
  • For More Information
• HTTP-NG
  • HTTP’s Growing Pains
  • HTTP-NG Activity
  • Modularize and Enhance
  • Distributed Objects
  • Layer 1: Messaging
  • Layer 2: Remote Invocation
  • Layer 3: Web Application
  • WebMUX
  • Binary Wire Protocol
  • Current Status
  • For More Information
• Part III
• Client Identification and Cookies
  • The Personal Touch
  • HTTP Headers
  • Client IP Address
  • User Login
  • Fat URLs
  • Cookies
    • Types of Cookies
    • How Cookies Work
    • Cookie Jar: Client-Side State
      • Netscape Navigator cookies
      • Microsoft Internet Explorer cookies
    • Different Cookies for Different Sites
      • Cookie Domain attribute
      • Cookie Path attribute
    • Cookie Ingredients
    • Version 0 (Netscape) Cookies
      • Version 0 Set-Cookie header
      • Version 0 Cookie header
    • Version 1 (RFC 2965) Cookies
      • Version 1 Set-Cookie2 header
      • Version 1 Cookie header
      • Version 1 Cookie2 header and version negotiation
    • Cookies and Session Tracking
    • Cookies and Caching
    • Cookies, Security, and Privacy
  • For More Information
• Basic Authentication
  • Authentication
    • HTTP’s Challenge/Response Authentication Framework
    • Authentication Protocols and Headers
    • Security Realms
  • Basic Authentication
    • Basic Authentication Example
    • Base-64 Username/Password Encoding
    • Proxy Authentication
  • The Security Flaws of Basic Authentication
  • For More Information
• Digest Authentication
  • The Improvements of Digest Authentication
    • Using Digests to Keep Passwords Secret
    • One-Way Digests
    • Using Nonces to Prevent Replays
    • The Digest Authentication Handshake
  • Digest Calculations
    • Digest Algorithm Input Data
    • The Algorithms H(d) and KD(s,d)
    • The Security-Related Data (A1)
    • The Message-Related Data (A2)
    • Overall Digest Algorithm
    • Digest Authentication Session
    • Preemptive Authorization
      • Next nonce pregeneration
      • Limited nonce reuse
      • Synchronized nonce generation
    • Nonce Selection
    • Symmetric Authentication
  • Quality of Protection Enhancements
    • Message Integrity Protection
    • Digest Authentication Headers
  • Practical Considerations
    • Multiple Challenges
    • Error Handling
    • Protection Spaces
    • Rewriting URIs
    • Caches
  • Security Considerations
    • Header Tampering
    • Replay Attacks
    • Multiple Authentication Mechanisms
    • Dictionary Attacks
    • Hostile Proxies and Man-in-the-Middle Attacks
    • Chosen Plaintext Attacks
    • Storing Passwords
  • For More Information
• Secure HTTP
  • Making HTTP Safe
    • HTTPS
  • Digital Cryptography
    • The Art and Science of Secret Coding
    • Ciphers
    • Cipher Machines
    • Keyed Ciphers
    • Digital Ciphers
  • Symmetric-Key Cryptography
    • Key Length and Enumeration Attacks
    • Establishing Shared Keys
  • Public-Key Cryptography
    • RSA
    • Hybrid Cryptosystems and Session Keys
  • Digital Signatures
    • Signatures Are Cryptographic Checksums
  • Digital Certificates
    • The Guts of a Certificate
    • X.509 v3 Certificates
    • Using Certificates to Authenticate Servers
  • HTTPS: The Details
    • HTTPS Overview
    • HTTPS Schemes
    • Secure Transport Setup
    • SSL Handshake
    • Server Certificates
    • Site Certificate Validation
    • Virtual Hosting and Certificates
  • A Real HTTPS Client
    • OpenSSL
    • A Simple HTTPS Client
    • Executing Our Simple OpenSSL Client
  • Tunneling Secure Traffic Through Proxies
  • For More Information
    • HTTP Security
    • SSL and TLS
    • Public-Key Infrastructure
    • Digital Cryptography
• Part IV
• Entities and Encodings
  • Messages Are Crates, Entities Are Cargo
    • Entity Bodies
  • Content-Length: The Entity’s Size
    • Detecting Truncation
    • Incorrect Content-Length
    • Content-Length and Persistent Connections
    • Content Encoding
    • Rules for Determining Entity Body Length
  • Entity Digests
  • Media Type and Charset
    • Character Encodings for Text Media
    • Multipart Media Types
    • Multipart Form Submissions
    • Multipart Range Responses
  • Content Encoding
    • The Content-Encoding Process
    • Content-Encoding Types
    • Accept-Encoding Headers
  • Transfer Encoding and Chunked Encoding
    • Safe Transport
    • Transfer-Encoding Headers
    • Chunked Encoding
      • Chunking and persistent connections
      • Trailers in chunked messages
    • Combining Content and Transfer Encodings
    • Transfer-Encoding Rules
  • Time-Varying Instances
  • Validators and Freshness
    • Freshness
    • Conditionals and Validators
  • Range Requests
  • Delta Encoding
    • Instance Manipulations, Delta Generators, and Delta Appliers
  • For More Information
• Internationalization
  • HTTP Support for International Content
  • Character Sets and HTTP
    • Charset Is a Character-to-Bits Encoding
    • How Character Sets and Encodings Work
    • The Wrong Charset Gives the Wrong Characters
    • Standardized MIME Charset Values
    • Content-Type Charset Header and META Tags
    • The Accept-Charset Header
  • Multilingual Character Encoding Primer
    • Character Set Terminology
    • Charset Is Poorly Named
    • Characters
    • Glyphs, Ligatures, and Presentation Forms
    • Coded Character Sets
      • US-ASCII: The mother of all character sets
      • iso-8859
      • JIS X 0201
      • JIS X 0208 and JIS X 0212
      • UCS
    • Character Encoding Schemes
      • 8-bit
      • UTF-8
      • iso-2022-jp
      • euc-jp
  • Language Tags and HTTP
    • The Content-Language Header
    • The Accept-Language Header
    • Types of Language Tags
    • Subtags
    • Capitalization
    • IANA Language Tag Registrations
    • First Subtag: Namespace
    • Second Subtag: Namespace
    • Remaining Subtags: Namespace
    • Configuring Language Preferences
    • Language Tag Reference Tables
  • Internationalized URIs
    • Global Transcribability Versus Meaningful Characters
    • URI Character Repertoire
    • Escaping and Unescaping
    • Escaping International Characters
    • Modal Switches in URIs
  • Other Considerations
    • Headers and Out-of-Spec Data
    • Dates
    • Domain Names
  • For More Information
    • Appendixes
    • Internet Internationalization
    • International Standards
• Content Negotiation and Transcoding
  • Content-Negotiation Techniques
  • Client-Driven Negotiation
  • Server-Driven Negotiation
    • Content-Negotiation Headers
    • Content-Negotiation Header Quality Values
    • Varying on Other Headers
    • Content Negotiation on Apache
      • Using type-map files
      • Using MultiViews
    • Server-Side Extensions
  • Transparent Negotiation
    • Caching and Alternates
    • The Vary Header
  • Transcoding
    • Format Conversion
    • Information Synthesis
    • Content Injection
    • Transcoding Versus Static Pregeneration
  • Next Steps
  • For More Information
• Part V
• Web Hosting
  • Hosting Services
    • A Simple Example: Dedicated Hosting
  • Virtual Hosting
    • Virtual Server Request Lacks Host Information
    • Making Virtual Hosting Work
      • Virtual hosting by URL path
      • Virtual hosting by port number
      • Virtual hosting by IP address
      • Virtual hosting by Host header
    • HTTP/1.1 Host Headers
      • Syntax and usage
      • Missing Host headers
      • Interpreting Host headers
      • Host headers and proxies
  • Making Web Sites Reliable
    • Mirrored Server Farms
    • Content Distribution Networks
    • Surrogate Caches in CDNs
    • Proxy Caches in CDNs
  • Making Web Sites Fast
  • For More Information
• Publishing Systems
  • FrontPage Server Extensions for Publishing Support
    • FrontPage Server Extensions
    • FrontPage Vocabulary
    • The FrontPage RPC Protocol
      • Request
      • Response
    • FrontPage Security Model
  • WebDAV and Collaborative Authoring
    • WebDAV Methods
    • WebDAV and XML
    • WebDAV Headers
    • WebDAV Locking and Overwrite Prevention
    • The LOCK Method
      • The opaquelocktoken scheme
      • The <lockdiscovery> XML element
      • Lock refreshes and the Timeout header
    • The UNLOCK Method
    • Properties and META Data
    • The PROPFIND Method
    • The PROPPATCH Method
    • Collections and Namespace Management
    • The MKCOL Method
    • The DELETE Method
    • The COPY and MOVE Methods
      • Overwrite header effect
      • COPY/MOVE of properties
      • Locked resources and COPY/MOVE
    • Enhanced HTTP/1.1 Methods
      • The PUT method
      • The OPTIONS method
    • Version Management in WebDAV
    • Future of WebDAV
  • For More Information
• Redirection and Load Balancing
  • Why Redirect?
  • Where to Redirect
  • Overview of Redirection Protocols
  • General Redirection Methods
    • HTTP Redirection
    • DNS Redirection
      • DNS round robin
      • Multiple addresses and round-robin address rotation
      • DNS round robin for load balancing
      • The impact of DNS caching
      • Other DNS-based redirection algorithms
    • Anycast Addressing
    • IP MAC Forwarding
    • IP Address Forwarding
    • Network Element Control Protocol
      • Messages
  • Proxy Redirection Methods
    • Explicit Browser Configuration
    • Proxy Auto-configuration
    • Web Proxy Autodiscovery Protocol
      • PAC file autodiscovery
      • WPAD algorithm
      • CURL discovery using DHCP
      • DNS A record lookup
      • Retrieving the PAC file
      • When to execute WPAD
      • WPAD spoofing
      • Timeouts
      • Administrator considerations
  • Cache Redirection Methods
    • WCCP Redirection
      • How WCCP redirection works
      • WCCP2 messages
      • Message components
      • Service groups
      • GRE packet encapsulation
      • WCCP load balancing
  • Internet Cache Protocol
  • Cache Array Routing Protocol
  • Hyper Text Caching Protocol
    • HTCP Authentication
    • Setting Caching Policies
  • For More Information
• Logging and Usage Tracking
  • What to Log?
  • Log Formats
    • Common Log Format
    • Combined Log Format
    • Netscape Extended Log Format
    • Netscape Extended 2 Log Format
    • Squid Proxy Log Format
  • Hit Metering
    • Overview
    • The Meter Header
  • A Word on Privacy
  • For More Information
• Part VI
• URI Schemes
• HTTP Status Codes
  • Status Code Classifications
  • Status Codes
• HTTP Header Reference
  • Accept
  • Accept-Charset
  • Accept-Encoding
  • Accept-Language
  • Accept-Ranges
  • Age
  • Allow
  • Authorization
  • Cache-Control
  • Client-ip
  • Connection
  • Content-Base
  • Content-Encoding
  • Content-Language
  • Content-Length
  • Content-Location
  • Content-MD5
  • Content-Range
  • Content-Type
  • Cookie
  • Cookie2
  • Date
  • ETag
  • Expect
  • Expires
  • From
  • Host
  • If-Modified-Since
  • If-Match
  • If-None-Match
  • If-Range
  • If-Unmodified-Since
  • Last-Modified
  • Location
  • Max-Forwards
  • MIME-Version
  • Pragma
  • Proxy-Authenticate
  • Proxy-Authorization
  • Proxy-Connection
  • Public
  • Range
  • Referer
  • Retry-After
  • Server
  • Set-Cookie
  • Set-Cookie2
  • TE
  • Trailer
  • Title
  • Transfer-Encoding
  • UA-(CPU, Disp, OS, Color, Pixels)
  • Upgrade
  • User-Agent
  • Vary
  • Via
  • Warning
  • WWW-Authenticate
  • X-Cache
  • X-Forwarded-For
  • X-Pad
  • X-Serial-Number
• MIME Types
  • Background
  • MIME Type Structure
    • Discrete Types
    • Composite Types
    • Multipart Types
    • Syntax
  • MIME Type IANA Registration
    • Registration Trees
    • Registration Process
    • Registration Rules
    • Registration Template
    • MIME Media Type Registry
  • MIME Type Tables
    • application/*
    • audio/*
    • chemical/*
    • image/*
    • message/*
    • model/*
    • multipart/*
    • text/*
    • video/*
    • Experimental Types
• Base-64 Encoding
  • Base-64 Encoding Makes Binary Data Safe
  • Eight Bits to Six Bits
  • Base-64 Padding
  • Perl Implementation
  • For More Information
• Digest Authentication
  • Digest WWW-Authenticate Directives
  • Digest Authorization Directives
  • Digest Authentication-Info Directives
  • Reference Code
    • File “digcalc.h”
    • File “digcalc.c”
    • File “digtest.c”
• Language Tags
  • First Subtag Rules
  • Second Subtag Rules
  • IANA-Registered Language Tags
  • ISO 639 Language Codes
  • ISO 3166 Country Codes
  • Language Administrative Organizations
• MIME Charset Registry
  • MIME Charset Registry
  • Preferred MIME Names
  • Registered Charsets
• Index