Security Engineering: A Guide To Building Dependable Distributed Systems Engineering

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 1083

DownloadSecurity Engineering: A Guide To Building Dependable Distributed Systems Engineering
Open PDF In BrowserView PDF
Security Engineering
A Guide to Building
Dependable Distributed
Systems
Second Edition
Ross J. Anderson

Wiley Publishing, Inc.

Security Engineering: A Guide to Building Dependable Distributed Systems,
Second Edition
Published by
Wiley Publishing, Inc.
10475 Crosspoint Boulevard
Indianapolis, IN 46256
Copyright © 2008 by Ross J. Anderson. All Rights Reserved.
Published by Wiley Publishing, Inc., Indianapolis, Indiana
Published simultaneously in Canada
ISBN: 978-0-470-06852-6
Manufactured in the United States of America
10 9 8 7 6 5 4 3 2 1
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections
107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or
authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood
Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be
addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317)
572-3447, fax (317) 572-4355, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with
respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including
without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or
promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work
is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional
services. If professional assistance is required, the services of a competent professional person should be sought.
Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or
Website is referred to in this work as a citation and/or a potential source of further information does not mean that
the author or the publisher endorses the information the organization or Website may provide or recommendations
it may make. Further, readers should be aware that Internet Websites listed in this work may have changed or
disappeared between when this work was written and when it is read.
For general information on our other products and services or to obtain technical support, please contact our Customer
Care Department within the U.S. at (800) 762-2974, outside the U.S. at (317) 572-3993 or fax (317) 572-4002.
Library of Congress Cataloging-in-Publication Data
Anderson, Ross, 1956Security engineering : a guide to building dependable distributed systems / Ross J Anderson. — 2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-06852-6 (cloth)
1. Computer security. 2. Electronic data processing–Distributed processing. I. Title.
QA76.9.A25A54 2008
005.1–dc22
2008006392
Trademarks: Wiley, the Wiley logo, and related trade dress are trademarks or registered trademarks of John Wiley
& Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written
permission. All other trademarks are the property of their respective owners. Wiley Publishing, Inc. is not associated
with any product or vendor mentioned in this book.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be
available in electronic books.

To Shireen

Credits

Executive Editor
Carol Long
Senior Development
Editor
Tom Dinse
Production Editor
Tim Tate
Editorial Manager
Mary Beth Wakefield
Production Manager
Tim Tate
Vice President
and Executive Group
Publisher
Richard Swadley

Vice President
and Executive
Publisher
Joseph B. Wikert
Project Coordinator,
Cover
Lynsey Stanford
Proofreader
Nancy Bell
Indexer
Jack Lewis
Cover Image
© Digital Vision/Getty Images
Cover Design
Michael E. Trent

v

Contents at a Glance

Preface to the Second Edition

xxv

Foreword by Bruce Schneier

xxvii

Preface

xxix

Acknowledgments

xxxv

Part I
Chapter 1

What Is Security Engineering?

3

Chapter 2

Usability and Psychology

17

Chapter 3

Protocols

63

Chapter 4

Access Control

93

Chapter 5

Cryptography

129

Chapter 6

Distributed Systems

185

Chapter 7

Economics

215

Chapter 8

Multilevel Security

239

Chapter 9

Multilateral Security

275

Part II

Chapter 10 Banking and Bookkeeping

313

Chapter 11 Physical Protection

365

Chapter 12 Monitoring and Metering

389

Chapter 13 Nuclear Command and Control

415
vii

viii

Contents at a Glance
Chapter 14 Security Printing and Seals

433

Chapter 15 Biometrics

457

Chapter 16 Physical Tamper Resistance

483

Chapter 17 Emission Security

523

Chapter 18 API Attacks

547

Chapter 19 Electronic and Information Warfare

559

Chapter 20 Telecom System Security

595

Chapter 21 Network Attack and Defense

633

Chapter 22 Copyright and DRM

679

Chapter 23 The Bleeding Edge

727

Part III
Chapter 24 Terror, Justice and Freedom

769

Chapter 25 Managing the Development of Secure Systems

815

Chapter 26 System Evaluation and Assurance

857

Chapter 27 Conclusions

889

Bibliography

893

Index

997

Contents

Preface to the Second Edition

xxv

Foreword by Bruce Schneier

xxvii

Preface

xxix

Acknowledgments

xxxv

Part I
Chapter 1

What Is Security Engineering?
Introduction
A Framework
Example 1–A Bank
Example 2–A Military Base
Example 3–A Hospital
Example 4–The Home
Definitions
Summary

3
3
4
6
7
9
10
11
15

Chapter 2

Usability and Psychology
Introduction
Attacks Based on Psychology
Pretexting
Phishing
Insights from Psychology Research
What the Brain Does Worse Than the Computer
Perceptual Bias and Behavioural Economics
Different Aspects of Mental Processing
Differences Between People
Social Psychology
What the Brain Does Better Than Computer

17
17
18
19
21
22
23
24
26
27
28
30
ix

x

Contents

Chapter 3

Passwords
Difficulties with Reliable Password Entry
Difficulties with Remembering the Password
Naive Password Choice
User Abilities and Training
Design Errors
Operational Issues
Social-Engineering Attacks
Trusted Path
Phishing Countermeasures
Password Manglers
Client Certs or Specialist Apps
Using the Browser’s Password Database
Soft Keyboards
Customer Education
Microsoft Passport
Phishing Alert Toolbars
Two-Factor Authentication
Trusted Computing
Fortified Password Protocols
Two-Channel Authentication
The Future of Phishing
System Issues
Can You Deny Service?
Protecting Oneself or Others?
Attacks on Password Entry
Interface Design
Eavesdropping
Technical Defeats of Password Retry Counters
Attacks on Password Storage
One-Way Encryption
Password Cracking
Absolute Limits
CAPTCHAs
Summary
Research Problems
Further Reading

31
32
33
34
35
37
39
40
42
43
43
44
44
45
45
46
47
47
48
49
49
50
52
53
53
54
54
55
55
56
56
57
57
59
60
61
61

Protocols
Introduction
Password Eavesdropping Risks
Who Goes There? — Simple Authentication
Challenge and Response
The MIG-in-the-Middle Attack
Reflection Attacks
Manipulating the Message
Changing the Environment

63
63
65
66
70
73
76
78
79

Contents
Chosen Protocol Attacks
Managing Encryption Keys
Basic Key Management
The Needham-Schroeder Protocol
Kerberos
Practical Key Management
Getting Formal
A Typical Smartcard Banking Protocol
The BAN Logic
Verifying the Payment Protocol
Limitations of Formal Verification
Summary
Research Problems
Further Reading
Chapter 4

Access Control
Introduction
Operating System Access Controls
Groups and Roles
Access Control Lists
Unix Operating System Security
Apple’s OS/X
Windows — Basic Architecture
Capabilities
Windows — Added Features
Middleware
Database Access Controls
General Middleware Issues
ORBs and Policy Languages
Sandboxing and Proof-Carrying Code
Virtualization
Trusted Computing
Hardware Protection
Intel Processors, and ‘Trusted Computing’
ARM Processors
Security Processors
What Goes Wrong
Smashing the Stack
Other Technical Attacks
User Interface Failures
Why So Many Things Go Wrong
Remedies
Environmental Creep
Summary
Research Problems
Further Reading

80
82
83
84
85
86
87
87
88
89
90
91
92
92
93
93
96
98
99
100
101
102
103
104
107
107
108
109
110
111
111
113
114
116
116
117
118
119
121
122
124
125
126
127
127

xi

xii

Contents
Chapter 5

Cryptography
Introduction
Historical Background
An Early Stream Cipher — The Vigenère
The One-Time Pad
An Early Block Cipher — Playfair
One-Way Functions
Asymmetric Primitives
The Random Oracle Model
Random Functions — Hash Functions
Properties
The Birthday Theorem
Random Generators — Stream Ciphers
Random Permutations — Block Ciphers
Public Key Encryption and Trapdoor One-Way Permutations
Digital Signatures
Symmetric Crypto Primitives
SP-Networks
Block Size
Number of Rounds
Choice of S-Boxes
Linear Cryptanalysis
Differential Cryptanalysis
Serpent
The Advanced Encryption Standard (AES)
Feistel Ciphers
The Luby-Rackoff Result
DES
Modes of Operation
Electronic Code Book
Cipher Block Chaining
Output Feedback
Counter Encryption
Cipher Feedback
Message Authentication Code
Composite Modes of Operation
Hash Functions
Extra Requirements on the Underlying Cipher
Common Hash Functions and Applications
Asymmetric Crypto Primitives
Cryptography Based on Factoring
Cryptography Based on Discrete Logarithms
Public Key Encryption — Diffie Hellman and ElGamal
Key Establishment
Digital Signature
Special Purpose Primitives

129
129
130
131
132
134
136
138
138
140
141
142
143
144
146
147
149
149
150
150
151
151
152
153
153
155
157
157
160
160
161
161
162
163
163
164
165
166
167
170
170
173
174
175
176
178

Contents
Elliptic Curve Cryptography
Certification
The Strength of Asymmetric Cryptographic Primitives

179
179
181

Summary
Research Problems
Further Reading

182
183
183

Chapter 6

Distributed Systems
Introduction
Concurrency
Using Old Data Versus Paying to Propagate State
Locking to Prevent Inconsistent Updates
The Order of Updates
Deadlock
Non-Convergent State
Secure Time
Fault Tolerance and Failure Recovery
Failure Models
Byzantine Failure
Interaction with Fault Tolerance
What Is Resilience For?
At What Level Is the Redundancy?
Service-Denial Attacks
Naming
The Distributed Systems View of Naming
What Else Goes Wrong
Naming and Identity
Cultural Assumptions
Semantic Content of Names
Uniqueness of Names
Stability of Names and Addresses
Adding Social Context to Naming
Restrictions on the Use of Names
Types of Name
Summary
Research Problems
Further Reading

185
185
186
186
188
188
189
190
191
192
193
193
194
195
197
198
200
200
204
204
206
207
207
208
209
210
211
211
212
213

Chapter 7

Economics
Introduction
Classical Economics
Monopoly
Public Goods
Information Economics
The Price of Information
The Value of Lock-In
Asymmetric Information

215
215
216
217
219
220
220
221
223

xiii

xiv

Contents
Game Theory
The Prisoners’ Dilemma
Evolutionary Games
The Economics of Security and Dependability
Weakest Link, or Sum of Efforts?
Managing the Patching Cycle
Why Is Windows So Insecure?
Economics of Privacy
Economics of DRM
Summary
Research Problems
Further Reading

223
225
226
228
229
229
230
232
233
234
235
235

Multilevel Security
Introduction
What Is a Security Policy Model?
The Bell-LaPadula Security Policy Model
Classifications and Clearances
Information Flow Control
The Standard Criticisms of Bell-LaPadula
Alternative Formulations
The Biba Model and Vista
Historical Examples of MLS Systems
SCOMP
Blacker
MLS Unix and Compartmented Mode Workstations
The NRL Pump
Logistics Systems
Sybard Suite
Wiretap Systems
Future MLS Systems
Vista
Linux
Virtualization
Embedded Systems
What Goes Wrong
Composability
The Cascade Problem
Covert Channels
The Threat from Viruses
Polyinstantiation
Other Practical Problems
Broader Implications of MLS

239
239
240
242
243
245
246
248
250
252
252
253
253
254
255
256
256
257
257
258
260
261
261
261
262
263
265
266
267
269

Part II
Chapter 8

Contents

Chapter 9

Summary
Research Problems
Further Reading

272
272
272

Multilateral Security
Introduction
Compartmentation, the Chinese Wall and the BMA Model
Compartmentation and the Lattice Model
The Chinese Wall
The BMA Model
The Threat Model
The Security Policy
Pilot Implementations
Current Privacy Issues
Inference Control
Basic Problems of Inference Control in Medicine
Other Applications of Inference Control
The Theory of Inference Control
Query Set Size Control
Trackers
More Sophisticated Query Controls
Cell Suppression
Maximum Order Control and the Lattice Model
Audit Based Control
Randomization
Limitations of Generic Approaches
Active Attacks
The Value of Imperfect Protection
The Residual Problem
Summary
Research Problems
Further Reading

275
275
277
277
281
282
284
287
289
290
293
293
296
297
298
298
298
299
300
300
301
302
304
305
306
309
310
310

Chapter 10 Banking and Bookkeeping
Introduction
The Origins of Bookkeeping
Double-Entry Bookkeeping
A Telegraphic History of E-commerce
How Bank Computer Systems Work
The Clark-Wilson Security Policy Model
Designing Internal Controls
What Goes Wrong
Wholesale Payment Systems
SWIFT
What Goes Wrong
Automatic Teller Machines
ATM Basics

313
313
315
316
316
317
319
320
324
328
329
331
333
334

xv

xvi

Contents
What Goes Wrong
Incentives and Injustices

Credit Cards
Fraud
Forgery
Automatic Fraud Detection
The Economics of Fraud
Online Credit Card Fraud — the Hype and the Reality
Smartcard-Based Banking
EMV
Static Data Authentication
Dynamic Data Authentication
Combined Data Authentication
RFID
Home Banking and Money Laundering
Summary
Research Problems
Further Reading

337
341

343
344
345
346
347
348
350
351
352
356
356
357
358
361
362
363

Chapter 11 Physical Protection
Introduction
Threats and Barriers
Threat Model
Deterrence
Walls and Barriers
Mechanical Locks
Electronic Locks
Alarms
How not to Protect a Painting
Sensor Defeats
Feature Interactions
Attacks on Communications
Lessons Learned
Summary
Research Problems
Further Reading

365
365
366
367
368
370
372
376
378
379
380
382
383
386
387
388
388

Chapter 12 Monitoring and Metering
Introduction
Prepayment Meters
Utility Metering
How the System Works
What Goes Wrong
Taxi Meters, Tachographs and Truck Speed Limiters
The Tachograph
What Goes Wrong
How Most Tachograph Manipulation Is Done

389
389
390
392
393
395
397
398
399
400

Contents
Tampering with the Supply
Tampering with the Instrument
High-Tech Attacks
The Digital Tachograph Project
System Level Problems
Other Problems
The Resurrecting Duckling

Postage Meters
Summary
Research Problems
Further Reading

401
401
402
403
404
405
407

408
412
413
414

Chapter 13 Nuclear Command and Control
Introduction
The Evolution of Command and Control
The Kennedy Memorandum
Authorization, Environment, Intent
Unconditionally Secure Authentication
Shared Control Schemes
Tamper Resistance and PALs
Treaty Verification
What Goes Wrong
Secrecy or Openness?
Summary
Research Problems
Further Reading

415
415
417
418
419
420
422
424
426
427
429
430
430
430

Chapter 14 Security Printing and Seals
Introduction
History
Security Printing
Threat Model
Security Printing Techniques
Packaging and Seals
Substrate Properties
The Problems of Glue
PIN Mailers
Systemic Vulnerabilities
Peculiarities of the Threat Model
Anti-Gundecking Measures
The Effect of Random Failure
Materials Control
Not Protecting the Right Things
The Cost and Nature of Inspection
Evaluation Methodology
Summary
Research Problems
Further Reading

433
433
434
435
436
437
443
443
444
445
446
447
448
449
450
451
451
453
454
454
455

xvii

xviii Contents
Chapter 15 Biometrics
Introduction
Handwritten Signatures
Face Recognition
Bertillonage
Fingerprints
Verifying Positive or Negative Identity Claims
Crime Scene Forensics
Iris Codes
Voice Recognition
Other Systems
What Goes Wrong
Summary
Research Problems
Further Reading

457
457
458
461
464
464
466
469
472
475
476
477
481
482
482

Chapter 16 Physical Tamper Resistance
Introduction
History
High-End Physically Secure Processors
Evaluation
Medium Security Processors
The iButton
The Dallas 5000 Series
FPGA Security, and the Clipper Chip
Smartcards and Microcontrollers
History
Architecture
Security Evolution
The State of the Art
Defense in Depth
Stop Loss
What Goes Wrong
The Trusted Interface Problem
Conflicts
The Lemons Market, Risk Dumping and Evaluation
Security-By-Obscurity
Interaction with Policy
Function Creep
So What Should One Protect?
Summary
Research Problems
Further Reading

483
483
485
486
492
494
494
495
496
499
500
501
501
512
513
513
514
514
515
516
517
517
518
518
520
520
520

Chapter 17 Emission Security
Introduction
History

523
523
524

Contents
Technical Surveillance and Countermeasures
Passive Attacks
Leakage Through Power and Signal Cables
Red/Black Separation
Timing Analysis
Power Analysis
Leakage Through RF Signals
Active Attacks
Tempest Viruses
Nonstop
Glitching
Differential Fault Analysis
Combination Attacks
Commercial Exploitation
Defenses
Optical, Acoustic and Thermal Side Channels
How Serious are Emsec Attacks?
Governments
Businesses
Summary
Research Problems
Further Reading

526
530
530
530
531
531
534
538
538
539
540
540
540
541
541
542
544
544
545
546
546
546

Chapter 18 API Attacks
Introduction
API Attacks on Security Modules
The XOR-To-Null-Key Attack
The Attack on the 4758
Multiparty Computation, and Differential Protocol Attacks
The EMV Attack
API Attacks on Operating Systems
Summary
Research Problems
Further Reading

547
547
548
549
551
552
553
554
555
557
557

Chapter 19 Electronic and Information Warfare
Introduction
Basics
Communications Systems
Signals Intelligence Techniques
Attacks on Communications
Protection Techniques
Frequency Hopping
DSSS
Burst Communications
Combining Covertness and Jam Resistance
Interaction Between Civil and Military Uses

559
559
560
561
563
565
567
568
569
570
571
572

xix

xx

Contents
Surveillance and Target Acquisition
Types of Radar
Jamming Techniques
Advanced Radars and Countermeasures
Other Sensors and Multisensor Issues
IFF Systems
Improvised Explosive Devices
Directed Energy Weapons
Information Warfare
Definitions
Doctrine
Potentially Useful Lessons from Electronic Warfare
Differences Between E-war and I-war
Summary
Research Problems
Further Reading

574
574
575
577
578
579
582
584
586
587
588
589
591
592
592
593

Chapter 20 Telecom System Security
Introduction
Phone Phreaking
Attacks on Metering
Attacks on Signaling
Attacks on Switching and Configuration
Insecure End Systems
Feature Interaction
Mobile Phones
Mobile Phone Cloning
GSM Security Mechanisms
Third Generation Mobiles — 3gpp
Platform Security
So Was Mobile Security a Success or a Failure?
VOIP
Security Economics of Telecomms
Frauds by Phone Companies
Billing Mechanisms
Summary
Research Problems
Further Reading

595
595
596
596
599
601
603
605
606
607
608
617
619
621
623
624
625
627
630
631
632

Chapter 21 Network Attack and Defense
Introduction
Vulnerabilities in Network Protocols
Attacks on Local Networks
Attacks Using Internet Protocols and Mechanisms
SYN Flooding
Smurfing
Distributed Denial of Service Attacks

633
633
635
636
638
638
639
640

Contents
Spam
DNS Security and Pharming

Trojans, Viruses, Worms and Rootkits
Early History of Malicious Code
The Internet Worm
How Viruses and Worms Work
The History of Malware
Countermeasures
Defense Against Network Attack
Configuration Management and Operational Security
Filtering: Firewalls, Spam Filters, Censorware and Wiretaps
Packet Filtering
Circuit Gateways
Application Relays
Ingress Versus Egress Filtering
Architecture
Intrusion Detection
Types of Intrusion Detection
General Limitations of Intrusion Detection
Specific Problems Detecting Network Attacks
Encryption
SSH
WiFi
Bluetooth
HomePlug
IPsec
TLS
PKI
Topology
Summary
Research Problems
Further Reading
Chapter 22 Copyright and DRM
Introduction
Copyright
Software
Books
Audio
Video and Pay-TV
Typical System Architecture
Video Scrambling Techniques
Attacks on Hybrid Scrambling Systems
DVB
DVD
HD-DVD and Blu-ray
AACS — Broadcast Encryption and Traitor Tracing

642
643

644
644
645
646
647
650
652
652
654
654
655
655
657
657
660
661
662
664
665
665
666
668
668
669
670
672
675
676
677
678
679
679
680
681
688
689
690
690
691
693
697
698
701
701

xxi

xxii

Contents
Blu-ray and SPDC
General Platforms
Windows Media Rights Management
Other Online Rights-Management Systems
Peer-to-Peer Systems
Rights Management of Semiconductor IP
Information Hiding
Watermarks and Copy Generation Management
General Information Hiding Techniques
Attacks on Copyright Marking Schemes
Applications of Copyright Marking Schemes
Policy
The IP Lobby
Who Benefits?
Accessory Control
Summary
Research Problems
Further Reading

Chapter 23 The Bleeding Edge
Introduction
Computer Games
Types of Cheating
Aimbots and Other Unauthorized Software
Virtual Worlds, Virtual Economies
Web Applications
eBay
Google
Social Networking Sites
Privacy Technology
Anonymous Email — The Dining Cryptographers and Mixes
Anonymous Web Browsing — Tor
Confidential and Anonymous Phone Calls
Email Encryption
Steganography and Forensics Countermeasures
Putting It All Together
Elections
Summary
Research Problems
Further Reading

703
704
705
706
707
709
710
711
712
714
718
718
720
722
723
725
725
726

727
727
728
730
732
733
734
735
736
739
745
747
749
751
753
755
757
759
764
764
765

Part III
Chapter 24 Terror, Justice and Freedom
Introduction
Terrorism
Causes of Political Violence

769
769
771
772

Contents xxiii
The Psychology of Political Violence
The Role of Political Institutions
The Role of the Press
The Democratic Response

Surveillance
The History of Government Wiretapping
The Growing Controversy about Traffic Analysis
Unlawful Surveillance
Access to Search Terms and Location Data
Data Mining
Surveillance via ISPs — Carnivore and its Offspring
Communications Intelligence on Foreign Targets
Intelligence Strengths and Weaknesses
The Crypto Wars
The Back Story to Crypto Policy
DES and Crypto Research
The Clipper Chip
Did the Crypto Wars Matter?
Export Control
Censorship
Censorship by Authoritarian Regimes
Network Neutrality
Peer-to-Peer, Hate Speech and Child Porn
Forensics and Rules of Evidence
Forensics
Admissibility of Evidence
Privacy and Data Protection
European Data Protection
Differences between Europe and the USA
Summary
Research Problems
Further Reading
Chapter 25 Managing the Development of Secure Systems
Introduction
Managing a Security Project
A Tale of Three Supermarkets
Risk Management
Organizational Issues
The Complacency Cycle and the Risk Thermostat
Interaction with Reliability
Solving the Wrong Problem
Incompetent and Inexperienced Security Managers
Moral Hazard
Methodology
Top-Down Design
Iterative Design

772
774
775
775

776
776
779
781
782
783
784
785
787
789
790
792
793
794
796
797
798
800
801
803
803
806
808
809
810
812
813
813
815
815
816
816
818
819
820
821
822
823
823
824
826
827

xxiv

Contents
Lessons from Safety-Critical Systems
Security Requirements Engineering
Managing Requirements Evolution
Bug Fixing
Control Tuning and Corporate Governance
Evolving Environments and the Tragedy of the Commons
Organizational Change
Managing Project Requirements
Parallelizing the Process
Risk Management
Managing the Team
Summary
Research Problems
Further Reading

829
834
835
836
838
839
841
842
844
846
848
852
853
854

Chapter 26 System Evaluation and Assurance
Introduction
Assurance
Perverse Economic Incentives
Project Assurance
Security Testing
Formal Methods
Quis Custodiet?
Process Assurance
Assurance Growth
Evolution and Security Assurance
Evaluation
Evaluations by the Relying Party
The Common Criteria
What the Common Criteria Don’t Do
Corruption, Manipulation and Inertia
Ways Forward
Hostile Review
Free and Open-Source Software
Semi-Open Design
Penetrate-and-Patch, CERTs, and Bugtraq
Education
Summary
Research Problems
Further Reading

857
857
858
858
860
861
862
862
863
866
868
869
870
873
876
878
881
882
882
884
885
886
887
887
887

Chapter 27 Conclusions

889

Bibliography

893

Index

997

Preface to the Second Edition

The first edition of Security Engineering was published in May 2001. Since then
the world has changed.
System security was one of Microsoft’s lowest priorities then; it’s now one
of the highest. The volume of malware continues to increase along with the
nuisance that it causes. Although a lot of effort has gone into defence — we
have seen Windows NT replaced by XP and then Vista, and occasional service
packs replaced by monthly security patches — the effort put into attacks has
increased far more. People who write viruses no longer do so for fun, but for
profit; the last few years have seen the emergence of a criminal economy that
supports diverse specialists. Spammers, virus writers, phishermen, money
launderers and spies trade busily with each other.
Cryptography has also moved on. The Advanced Encryption Standard is
being embedded into more and more products, and we have some interesting
developments on the public-key side of things too. But just as our algorithm
problems get solved, so we face a host of implementation issues. Side channels,
poorly designed APIs and protocol failures continue to break systems. Applied
cryptography is harder than ever to do well.
Pervasive computing also opens up new challenges. As computers and
communications become embedded invisibly everywhere, so problems that
used to only afflict ‘proper computers’ crop up in all sorts of other devices too.
What does it mean for a thermometer to be secure, or an air-conditioner?
The great diversity of intelligent devices brings with it a great diversity
of interests and actors. Security is not just about keeping the bad guys out,
but increasingly concerned with tussles for power and control. DRM pits the
content and platform industries against consumers, and against each other;
accessory control is used to tie printers to their vendors’ cartridges, but leads
xxv

xxvi

Preface to the Second Edition

to antitrust lawsuits and government intervention. Security also interacts with
safety in applications from cars through utilities to electronic healthcare. The
security engineer needs to understand not just crypto and operating systems,
but economics and human factors as well.
And the ubiquity of digital devices means that ‘computer security’ is no
longer just a problem for a few systems specialists. Almost all white-collar
crime (and much crime of the serious violent sort) now involves computers
or mobile phones, so a detective needs to understand computer forensics just
as she needs to know how to drive. More and more lawyers, accountants,
managers and other people with no formal engineering training are going to
have to understand system security in order to do their jobs well.
The rapid growth of online services, from Google and Facebook to massively
multiplayer games, has also changed the world. Bugs in online applications
can be fixed rapidly once they’re noticed, but the applications get ever more
complex and their side-effects harder to predict. We may have a reasonably
good idea what it means for an operating system or even a banking service to
be secure, but we can’t make any such claims for online lifestyles that evolve
all the time. We’re entering a novel world of evolving socio-technical systems,
and that raises profound questions about how the evolution is driven and who
is in control.
The largest changes, however, may be those driven by the tragic events of
September 2001 and by our reaction to them. These have altered perceptions
and priorities in many ways, and changed the shape of the security industry.
Terrorism is not just about risk, but about the perception of risk, and about
the manipulation of perception. This adds psychology and politics to the mix.
Security engineers also have a duty to contribute to the political debate. Where
inappropriate reactions to terrorist crimes have led to major waste of resources
and unforced policy errors, we have to keep on educating people to ask a
few simple questions: what are we seeking to prevent, and will the proposed
mechanisms actually work?
Ross Anderson
Cambridge, January 2008

Foreword

In a paper he wrote with Roger Needham, Ross Anderson coined the phrase
‘‘programming Satan’s computer’’ to describe the problems faced by computersecurity engineers. It’s the sort of evocative image I’ve come to expect from
Ross, and a phrase I’ve used ever since.
Programming a computer is straightforward: keep hammering away at the
problem until the computer does what it’s supposed to do. Large application
programs and operating systems are a lot more complicated, but the methodology is basically the same. Writing a reliable computer program is much
harder, because the program needs to work even in the face of random errors
and mistakes: Murphy’s computer, if you will. Significant research has gone
into reliable software design, and there are many mission-critical software
applications that are designed to withstand Murphy’s Law.
Writing a secure computer program is another matter entirely. Security
involves making sure things work, not in the presence of random faults, but in
the face of an intelligent and malicious adversary trying to ensure that things
fail in the worst possible way at the worst possible time . . . again and again. It
truly is programming Satan’s computer.
Security engineering is different from any other kind of programming. It’s
a point I made over and over again: in my own book, Secrets and Lies, in
my monthly newsletter Crypto-Gram, and in my other writings. And it’s a
point Ross makes in every chapter of this book. This is why, if you’re doing
any security engineering . . . if you’re even thinking of doing any security
engineering, you need to read this book. It’s the first, and only, end-to-end
modern security design and engineering book ever written.
And it comes just in time. You can divide the history of the Internet
into three waves. The first wave centered around mainframes and terminals.
xxvii

xxviii Foreword

Computers were expensive and rare. The second wave, from about 1992 until
now, centered around personal computers, browsers, and large application
programs. And the third, starting now, will see the connection of all sorts
of devices that are currently in proprietary networks, standalone, and noncomputerized. By 2003, there will be more mobile phones connected to the
Internet than computers. Within a few years we’ll see many of the world’s
refrigerators, heart monitors, bus and train ticket dispensers, burglar alarms,
and electricity meters talking IP. Personal computers will be a minority player
on the Internet.
Security engineering, especially in this third wave, requires you to think
differently. You need to figure out not how something works, but how
something can be made to not work. You have to imagine an intelligent
and malicious adversary inside your system (remember Satan’s computer),
constantly trying new ways to subvert it. You have to consider all the ways
your system can fail, most of them having nothing to do with the design itself.
You have to look at everything backwards, upside down, and sideways. You
have to think like an alien.
As the late great science fiction editor John W. Campbell, said: ‘‘An alien
thinks as well as a human, but not like a human.’’ Computer security is a lot
like that. Ross is one of those rare people who can think like an alien, and then
explain that thinking to humans. Have fun reading.
Bruce Schneier
January 2001

Preface

For generations, people have defined and protected their property and their
privacy using locks, fences, signatures, seals, account books, and meters. These
have been supported by a host of social constructs ranging from international
treaties through national laws to manners and customs.
This is changing, and quickly. Most records are now electronic, from
bank accounts to registers of real property; and transactions are increasingly
electronic, as shopping moves to the Internet. Just as important, but less
obvious, are the many everyday systems that have been quietly automated.
Burglar alarms no longer wake up the neighborhood, but send silent messages
to the police; students no longer fill their dormitory washers and dryers with
coins, but credit them using a smartcard they recharge at the college bookstore;
locks are no longer simple mechanical affairs, but are operated by electronic
remote controls or swipe cards; and instead of renting videocassettes, millions
of people get their movies from satellite or cable channels. Even the humble
banknote is no longer just ink on paper, but may contain digital watermarks
that enable many forgeries to be detected by machine.
How good is all this new security technology? Unfortunately, the honest
answer is ‘nowhere near as good as it should be’. New systems are often rapidly
broken, and the same elementary mistakes are repeated in one application after
another. It often takes four or five attempts to get a security design right, and
that is far too many.
The media regularly report security breaches on the Internet; banks fight
their customers over ‘phantom withdrawals’ from cash machines; VISA reports
huge increases in the number of disputed Internet credit card transactions;
satellite TV companies hound pirates who copy their smartcards; and law

xxix

xxx

Preface

enforcement agencies try to stake out territory in cyberspace with laws controlling the use of encryption. Worse still, features interact. A mobile phone
that calls the last number again if one of the keys is pressed by accident may
be just a minor nuisance — until someone invents a machine that dispenses
a can of soft drink every time its phone number is called. When all of a
sudden you find 50 cans of Coke on your phone bill, who is responsible, the
phone company, the handset manufacturer, or the vending machine operator?
Once almost every electronic device that affects your life is connected to the
Internet — which Microsoft expects to happen by 2010 — what does ‘Internet
security’ mean to you, and how do you cope with it?
As well as the systems that fail, many systems just don’t work well enough.
Medical record systems don’t let doctors share personal health information
as they would like, but still don’t protect it against inquisitive private eyes.
Zillion-dollar military systems prevent anyone without a ‘top secret’ clearance
from getting at intelligence data, but are often designed so that almost everyone
needs this clearance to do any work. Passenger ticket systems are designed to
prevent customers cheating, but when trustbusters break up the railroad, they
cannot stop the new rail companies cheating each other. Many of these failures
could have been foreseen if designers had just a little bit more knowledge of
what had been tried, and had failed, elsewhere.
Security engineering is the new discipline that is starting to emerge out of
all this chaos.
Although most of the underlying technologies (cryptology, software reliability, tamper resistance, security printing, auditing, etc.) are relatively well
understood, the knowledge and experience of how to apply them effectively
is much scarcer. And since the move from mechanical to digital mechanisms
is happening everywhere at once, there just has not been time for the lessons
learned to percolate through the engineering community. Time and again, we
see the same old square wheels being reinvented.
The industries that have managed the transition most capably are often
those that have been able to borrow an appropriate technology from another
discipline. Examples include the reuse of technology designed for military
identify-friend-or-foe equipment in bank cash machines and even prepayment
gas meters. So even if a security designer has serious expertise in some particular speciality — whether as a mathematician working with ciphers or a
chemist developing banknote inks — it is still prudent to have an overview
of the whole subject. The essence of good security engineering is understanding the potential threats to a system, then applying an appropriate mix
of protective measures — both technological and organizational — to control
them. Knowing what has worked, and more importantly what has failed, in
other applications is a great help in developing judgment. It can also save a lot
of money.

Preface

The purpose of this book is to give a solid introduction to security engineering, as we understand it at the beginning of the twenty-first century. My goal
is that it works at four different levels:
1. As a textbook that you can read from one end to the other over a few days as an
introduction to the subject. The book is to be used mainly by the working
IT professional who needs to learn about the subject, but it can also be
used in a one-semester course in a university.
2. As a reference book to which you can come for an overview of the workings of
some particular type of system. These systems include cash machines, taxi
meters, radar jammers, anonymous medical record databases, and so on.
3. As an introduction to the underlying technologies, such as crypto, access control, inference control, tamper resistance, and seals. Space prevents me from
going into great depth; but I provide a basic road map for each subject,
plus a reading list for the curious (and a list of open research problems
for the prospective graduate student).
4. As an original scientific contribution in which I have tried to draw out the common principles that underlie security engineering, and the lessons that people
building one kind of system should have learned from others. In the many
years I have been working in security, I keep coming across these. For
example, a simple attack on stream ciphers wasn’t known to the people
who designed a common antiaircraft fire control radar so it was easy
to jam; while a trick well known to the radar community wasn’t understood by banknote printers and people who design copyright marking
schemes, which led to a quite general attack on most digital watermarks.
I have tried to keep this book resolutely mid-Atlantic; a security engineering
book has to be, as many of the fundamental technologies are American, while
many of the interesting applications are European. (This isn’t surprising given
the better funding of U.S. universities and research labs, and the greater
diversity of nations and markets in Europe.) What’s more, many of the
successful European innovations — from the smart-card to the GSM mobile
phone to the pay-per-view TV service — have crossed the Atlantic and now
thrive in the Americas. Both the science, and the case studies, are necessary.
This book grew out of the security engineering courses I teach at Cambridge
University, but I have rewritten my notes to make them self-contained and
added at least as much material again. It should be useful to the established
professional security manager or consultant as a first-line reference; to the
computer science professor doing research in cryptology; to the working
police detective trying to figure out the latest computer scam; and to policy
wonks struggling with the conflicts involved in regulating cryptography and
anonymity. Above all, it is aimed at Dilbert. My main audience is the working

xxxi

xxxii Preface

programmer or engineer who is trying to design real systems that will keep on
working despite the best efforts of customers, managers, and everybody else.
This book is divided into three parts.
The first looks at basic concepts, starting with the central concept of a
security protocol, and going on to human-computer interface issues,
access controls, cryptology, and distributed system issues. It does not
assume any particular technical background other than basic computer
literacy. It is based on an Introduction to Security course that I teach to
second-year undergraduates.
The second part looks in much more detail at a number of important
applications, such as military communications, medical record systems,
cash machines, mobile phones, and pay-TV. These are used to introduce more of the advanced technologies and concepts. It also considers
information security from the viewpoint of a number of different interest groups, such as companies, consumers, criminals, police, and spies.
This material is drawn from my senior course on security, from research
work, and from experience consulting.
The third part looks at the organizational and policy issues: how computer security interacts with law, with evidence, and with corporate politics; how we can gain confidence that a system will perform as intended;
and how the whole business of security engineering can best be
managed.
I believe that building systems that continue to perform robustly in the face
of malice is one of the most important, interesting, and difficult tasks facing
engineers in the twenty-first century.
Ross Anderson
Cambridge, January 2001

About the Author

Why should I have been the person to write this book? Well, I seem to
have accumulated the right mix of experience and qualifications over the last
25 years. I graduated in mathematics and natural science from Cambridge
(England) in the 1970s, and got a qualification in computer engineering; my
first proper job was in avionics; and I became interested in cryptology and
computer security in the mid-1980s. After working in the banking industry for
several years, I started doing consultancy for companies that designed equipment for banks, and then working on other applications of this technology,
such as prepayment electricity meters.
I moved to academia in 1992, but continued to consult to industry on security
technology. During the 1990s, the number of applications that employed
cryptology rose rapidly: burglar alarms, car door locks, road toll tags, and
satellite TV encryption systems all made their appearance. As the first legal
disputes about these systems came along, I was lucky enough to be an expert
witness in some of the important cases. The research team I lead had the
good fortune to be in the right place at the right time when several crucial
technologies, such as tamper resistance and digital watermarking, became hot
topics.
By about 1996, it started to become clear to me that the existing textbooks
were too specialized. The security textbooks focused on the access control
mechanisms in operating systems, while the cryptology books gave very
detailed expositions of the design of cryptographic algorithms and protocols.
These topics are interesting, and important. However they are only part of
the story. Most system designers are not overly concerned with crypto or
operating system internals, but with how to use these tools effectively. They
are quite right in this, as the inappropriate use of mechanisms is one of the
main causes of security failure. I was encouraged by the success of a number
xxxiii

xxxiv About the Author

of articles I wrote on security engineering (starting with ‘Why Cryptosystems
Fail’ in 1993); and the need to teach an undergraduate class in security led to
the development of a set of lecture notes that made up about half of this book.
Finally, in 1999, I got round to rewriting them for a general technical audience.
I have learned a lot in the process; writing down what you think you know
is a good way of finding out what you don’t. I have also had a lot of fun. I
hope you have as much fun reading it!

Acknowledgments

A great many people have helped in various ways with this book. I probably
owe the greatest thanks to those who read the manuscript (or a large part of
it) looking for errors and obscurities. They were Anne Anderson, Ian Brown,
Nick Bohm, Richard Bondi, Caspar Bowden, Richard Clayton, Steve Early,
Rich Graveman, Markus Kuhn, Dan Lough, David MacKay, John McHugh,
Bob Morris, Roger Needham, Jerry Saltzer, Marv Schaefer, Karen Spärck Jones
and Frank Stajano. Much credit also goes to my editor, Carol Long, who
(among many other things) went through the first six chapters and coached
me on the style appropriate for a professional (as opposed to academic) book.
At the proofreading stage, I got quite invaluable help from Carola Bohm, Mike
Bond, Richard Clayton, George Danezis, and Bruce Godfrey.
A large number of subject experts also helped me with particular chapters
or sections. Richard Bondi helped me refine the definitions in Chapter 1;
Jianxin Yan, Alan Blackwell and Alasdair Grant helped me investigate the
applied psychology aspects of passwords; John Gordon and Sergei Skorobogatov were my main sources on remote key entry devices; Whit Diffie
and Mike Brown on IFF; Steve Early on Unix security (although some of my
material is based on lectures given by Ian Jackson); Mike Roe, Ian Kelly, Paul
Leyland, and Fabien Petitcolas on the security of Windows NT4 and Win2K;
Virgil Gligor on the history of memory overwriting attacks, and on mandatory
integrity policies; and Jean Bacon on distributed systems. Gary Graunke told
me the history of protection in Intel processors; Orr Dunkelman found many
bugs in a draft of the crypto chapter and John Brazier pointed me to the
Humpty Dumpty quote.
Moving to the second part of the book, the chapter on multilevel security was
much improved by input from Jeremy Epstein, Virgil Gligor, Jong-Hyeon Lee,
Ira Moskowitz, Paul Karger, Rick Smith, Frank Stajano, and Simon Wiseman,
xxxv

xxxvi

Acknowledgments

while Frank also helped with the following two chapters. The material on
medical systems was originally developed with a number of people at the
British Medical Association, most notably Fleur Fisher, Simon Jenkins, and
Grant Kelly. Denise Schmandt-Besserat taught the world about bullae, which
provided the background for the chapter on banking systems; that chapter
was also strengthened by input from Fay Hider and Willie List. The chapter
on alarms contains much that I was taught by Roger Needham, Peter Dean,
John Martin, Frank Clish, and Gary Geldart. Nuclear command and control
systems are much the brainchild of Gus Simmons; he and Bob Morris taught
me much of what’s in that chapter.
Sijbrand Spannenburg reviewed the chapter on security printing; and Roger
Johnston has taught us all an enormous amount about seals. John Daugman
helped polish the chapter on biometrics, as well as inventing iris scanning which I describe there. My tutors on tamper resistance were Oliver
Kömmerling and Markus Kuhn; Markus also worked with me on emission
security. I had substantial input on electronic warfare from Mike Brown and
Owen Lewis. The chapter on phone fraud owes a lot to Duncan Campbell,
Richard Cox, Rich Graveman, Udi Manber, Andrew Odlyzko and Roy Paterson. Ian Jackson contributed some ideas on network security. Fabien Petitcolas
‘wrote the book’ on copyright marking, and helped polish my chapter on it.
Johann Bezuidenhoudt made perceptive comments on both phone fraud and
electronic commerce, while Peter Landrock gave valuable input on bookkeeping and electronic commerce systems. Alistair Kelman was a fount of knowledge on the legal aspects of copyright; and Hal Varian kept me straight on matters of economics, and particularly the chapters on e-commerce and assurance.
As for the third part of the book, the chapter on e-policy was heavily influenced by colleagues at the Foundation for Information Policy Research, notably
Caspar Bowden, Nick Bohm, Fleur Fisher, Brian Gladman, Ian Brown, Richard
Clayton — and by the many others involved in the fight, including Whit Diffie,
John Gilmore, Susan Landau, Brian Omotani and Mark Rotenberg. The chapter
on management benefited from input from Robert Brady, Jack Lang, and Willie
List. Finally, my thinking on assurance has been influenced by many people,
including Robin Ball, Robert Brady, Willie List, and Robert Morris.
There were also many people over the years who taught me my trade. The
foremost of them is Roger Needham, who was my thesis advisor; but I also
learned a lot from hundreds of engineers, programmers, auditors, lawyers,
and policemen with whom I worked on various consultancy jobs over the last
15 years. Of course, I take the rap for all the remaining errors and omissions.
Finally, I owe a huge debt to my family, especially to my wife Shireen for
putting up with over a year in which I neglected household duties and was
generally preoccupied. Daughter Bavani and dogs Jimmy, Bess, Belle, Hobbes,
Bigfoot, Cat, and Dogmatix also had to compete for a diminished quantum of
attention, and I thank them for their forbearance.

Further Acknowledgments for
the Second Edition

Many of the folks who helped me with the first edition have also helped
update the same material this time. In addition, I’ve had useful input, feedback
or debugging assistance from Edmond Alyanakian, Johann Bezuidenhoudt,
Richard Clayton, Jolyon Clulow, Dan Cvrcek, Roger Dingledine, Saar Drimer,
Mike Ellims, Dan Geer, Gary Geldart, Wendy Grossman, Dan Hagon, Feng
Hao, Roger Johnston, Markus Kuhn, Susan Landau, Stephen Lewis, Nick
Mathewson, Tyler Moore, Steven Murdoch, Shishir Nagaraja, Roger Nebel,
Andy Ozment, Mike Roe, Frank Stajano, Mark Staples, Don Taylor, Marc
Tobias, Robert Watson and Jeff Yan. The members of our security group
in Cambridge, and the Advisory Council of the Foundation for Information
Policy Research, have been an invaluable sounding-board for many ideas. And
I am also grateful to the many readers of the first edition who pointed out
typos and other improvements: Piotr Carlson, Peter Chambers, Nick Drage,
Austin Donnelly, Ben Dougall, Shawn Fitzgerald, Paul Gillingwater, Pieter
Hartel, David Håsäther, Konstantin Hyppönen, Oliver Jorns, Markus Kuhn,
Garry McKay, Joe Osborne, Avi Rubin, Sam Simpson, M Taylor, Peter Taylor,
Paul Thomas, Nick Volenec, Randall Walker, Keith Willis, Stuart Wray and
Stefek Zaba.
A number of typos have been corrected in the second printing (2010). Thanks
to Adam Atkinson, Alastair Beresford, Antonomasia, David Boddie, Kristof
Boeynaems, Martin Brain, James Davenport, Dan Eble, Shailendra Fuloria,
Dan Hasather, Neil Jenkins, Hyoung Joong Kim, Patrick Koeberl, Simon
Kramer, Stephan Neuhaus, Mark Oeltjenbruns, Alexandros Papadopoulos,
Chris Pepper, Oscar Pereira, Raphael Phan, Matthew Slyman, Daniel WagnerHall, Randall Walker, and Stuart Wray for pointing them out!

xxxvii

Legal Notice

I cannot emphasize too strongly that the tricks taught in this book are intended
only to enable you to build better systems. They are not in any way given as
a means of helping you to break into systems, subvert copyright protection
mechanisms, or do anything else unethical or illegal.
Where possible I have tried to give case histories at a level of detail that
illustrates the underlying principles without giving a ‘hacker’s cookbook’.

Should This Book Be Published at All?
There are people who believe that the knowledge contained in this book
should not be published. This is an old debate; in previous centuries, people
objected to the publication of books on locksmithing, on the grounds that they
were likely to help the bad guys more than the good guys.
I think that these fears are answered in the first book in English that
discussed cryptology. This was a treatise on optical and acoustic telegraphy
written by Bishop John Wilkins in 1641 [805]. He traced scientific censorship
back to the Egyptian priests who forbade the use of alphabetic writing on the
grounds that it would spread literacy among the common people and thus
foster dissent. As he said:
It will not follow that everything must be suppresst which may be abused. . .
If all those useful inventions that are liable to abuse should therefore be
concealed there is not any Art or Science which may be lawfully profest.
The question was raised again in the nineteenth century, when some wellmeaning people wanted to ban books on locksmithing. A contemporary writer
on the subject replied [750]:
xxxix

xl

Legal Notice

Many well-meaning persons suppose that the discussion respecting the
means for baffling the supposed safety of locks offers a premium for
dishonesty, by showing others how to be dishonest. This is a fallacy.
Rogues are very keen in their profession, and already know much more
than we can teach them respecting their several kinds of roguery. Rogues
knew a good deal about lockpicking long before locksmiths discussed
it among themselves . . . if there be harm, it will be much more than
counterbalanced by good.
These views have been borne out by long experience since. As for me, I
worked for two separate banks for three and a half years on cash machine
security, but I learned significant new tricks from a document written by
a convicted card fraudster that circulated in the U.K. prison system. Many
government agencies are now coming round to this point of view. It is
encouraging to see, for example, that the U.S. National Security Agency has
published the specifications of the encryption algorithm (Skipjack) and the key
management protocol (KEA) used to protect secret U.S. government traffic.
Their judgment is clearly that the potential harm done by letting the Iraqis
use a decent encryption algorithm is less than the good that will be done by
having commercial off-the-shelf software compatible with Federal encryption
standards.
In short, while some bad guys will benefit from a book such as this, they
mostly know the tricks already, and the good guys will benefit much more.

PART

I

In this section of the book, I cover the basics of
security engineering technology. The first chapter sets
out to define the subject matter by giving an overview
of the secure distributed systems found in four environments: a bank, an air force base, a hospital, and the
home. The second chapter then plunges into the thick
of things by tackling usability. The interface between
the user and the machine is where the most intractable
problems lurk. Phishing is the most rapidly growing
online crime; pretexting is the main way in which
privacy is compromised; and psychology is likely to
be one of the most fruitful areas of security research
in coming years. We need to know more about how
people can be deceived, so we can design systems
that make deception harder. There is also the
problem that risk perceptions and realities have drifted
ever further apart, specially since 9/11.
The following chapters dig progressively deeper into
the technical meat. The third chapter is on security
protocols, which specify how the players in a system — whether people, computers, or other electronic
devices — communicate with each other. The fourth is
on access control: even once a client (be it a phone,
a PC, or whatever) has authenticated itself satisfactorily to a server, we still need mechanisms to control
which data it can read or write on the server and which
transactions it can execute. These mechanisms operate

2

Part I

at different levels — operating system, database, application — but share some
interesting characteristics and failure modes.
The fifth chapter is on the ‘duct tape’ that underlies most of the protocols
and holds distributed systems together: cryptography. This is the art (and
science) of codes and ciphers; it is much more than a clever means for keeping
messages secret from an eavesdropper. Nowadays its job is ‘taking trust from
where it exists to where it’s needed’ [853].
The next chapter is on distributed systems. Researchers in this field are
interested in topics such as concurrency control, fault tolerance, and naming.
These take on subtle new meanings when systems must be made resilient
against malice as well as against accidental failure. Using old data — replaying
old transactions or reusing the credentials of a user who has left some time
ago — is a serious problem, as is the multitude of names by which people are
known to different systems (email addresses, credit card numbers, subscriber
numbers, etc.). Many systems fail because their designers don’t appreciate
these issues.
The final chapter in this part is on economics. Security economics has grown
hugely since this book first appeared in 2001; we have come to realise that
many security failures are often due to perverse incentives rather than to the
lack of suitable technical protection mechanisms. (Indeed, the former often
explain the latter.) Security mechanisms are increasingly used not to keep
‘bad’ people out of ‘good’ systems, but to enable one principal to exert power
over another: examples are authentication mechanisms that compel you to
buy ink cartridges from your printer maker; and digital rights management
mechanisms that restrict the owner of a computer. (This was marketed as a
technology to protect the rights of the music industry, but in practice has turned
out to often maximise the income of the vendor of the rights-management
system). Ethical decisions are no longer a matter of ‘black hat’ versus ‘white
hat’, but can turn on whether the intended effect is anticompetitive. So the
modern security engineer needs to understand basic economic theory along
with the theories underlying ciphers, protocols and access controls.
Most of the material in these chapters is standard textbook fare, and the
chapters are intended to be pedagogic rather than encyclopaedic, so I have not
put in as many citations as in the rest of the book. I hope, however, that even
experts will find some of the case studies of value.

CHAPTER

1
What Is Security Engineering?
Out of the crooked timber of humanity, no straight
thing was ever made.
— Immanuel Kant
The world is never going to be perfect, either on- or offline; so
let’s not set impossibly high standards for online.
— Esther Dyson

1.1

Introduction

Security engineering is about building systems to remain dependable in the
face of malice, error, or mischance. As a discipline, it focuses on the tools,
processes, and methods needed to design, implement, and test complete
systems, and to adapt existing systems as their environment evolves.
Security engineering requires cross-disciplinary expertise, ranging from
cryptography and computer security through hardware tamper-resistance and
formal methods to a knowledge of economics, applied psychology, organizations and the law. System engineering skills, from business process analysis
through software engineering to evaluation and testing, are also important;
but they are not sufficient, as they deal only with error and mischance rather
than malice.
Many security systems have critical assurance requirements. Their failure
may endanger human life and the environment (as with nuclear safety and
control systems), do serious damage to major economic infrastructure (cash
machines and other bank systems), endanger personal privacy (medical record

3

4

Chapter 1

■

What Is Security Engineering?

systems), undermine the viability of whole business sectors (pay-TV), and
facilitate crime (burglar and car alarms). Even the perception that a system is
more vulnerable than it really is (paying with a credit card over the Internet)
can significantly hold up economic development.
The conventional view is that while software engineering is about ensuring that certain things happen (‘John can read this file’), security is about
ensuring that they don’t (‘The Chinese government can’t read this file’). Reality is much more complex. Security requirements differ greatly from one
system to another. One typically needs some combination of user authentication, transaction integrity and accountability, fault-tolerance, message secrecy,
and covertness. But many systems fail because their designers protect the
wrong things, or protect the right things but in the wrong way.
Getting protection right thus depends on several different types of process.
You have to figure out what needs protecting, and how to do it. You also
need to ensure that the people who will guard the system and maintain it are
properly motivated. In the next section, I’ll set out a framework for thinking
about this. Then, in order to illustrate the range of different things that security
systems have to do, I will take a quick look at four application areas: a bank,
an air force base, a hospital, and the home. Once we have given some concrete
examples of the stuff that security engineers have to understand and build, we
will be in a position to attempt some definitions.

1.2

A Framework

Good security engineering requires four things to come together. There’s
policy: what you’re supposed to achieve. There’s mechanism: the ciphers,
access controls, hardware tamper-resistance and other machinery that you
assemble in order to implement the policy. There’s assurance: the amount of
reliance you can place on each particular mechanism. Finally, there’s incentive:
the motive that the people guarding and maintaining the system have to do
their job properly, and also the motive that the attackers have to try to defeat
your policy. All of these interact (see Fig. 1.1).
As an example, let’s think of the 9/11 terrorist attacks. The hijackers’ success
in getting knives through airport security was not a mechanism failure but a
policy one; at that time, knives with blades up to three inches were permitted,
and the screeners did their task of keeping guns and explosives off as far as
we know. Policy has changed since then: first to prohibit all knives, then most
weapons (baseball bats are now forbidden but whiskey bottles are OK); it’s
flip-flopped on many details (butane lighters forbidden then allowed again).
Mechanism is weak, because of things like composite knives and explosives
that don’t contain nitrogen. Assurance is always poor; many tons of harmless
passengers’ possessions are consigned to the trash each month, while well

1.2 A Framework

Policy







Mechanism

 Incentives




















 Assurance

Figure 1.1: Security Engineering Analysis Framework

below half of all the weapons taken through screening (whether accidentally
or for test purposes) are picked up.
Serious analysts point out major problems with priorities. For example, the
TSA has spent $14.7 billion on aggressive passenger screening, which is fairly
ineffective, while $100 m spent on reinforcing cockpit doors would remove
most of the risk [1024]. The President of the Airline Pilots Security Alliance
notes that most ground staff aren’t screened, and almost no care is taken to
guard aircraft parked on the ground overnight. As most airliners don’t have
locks, there’s not much to stop a bad guy wheeling steps up to a plane and
placing a bomb on board; if he had piloting skills and a bit of chutzpah, he
could file a flight plan and make off with it [820]. Yet screening staff and
guarding planes are just not a priority.
Why are such poor policy choices made? Quite simply, the incentives on
the decision makers favour visible controls over effective ones. The result is
what Bruce Schneier calls ‘security theatre’ — measures designed to produce a
feeling of security rather than the reality. Most players also have an incentive to
exaggerate the threat from terrorism: politicians to scare up the vote, journalists
to sell more papers, companies to sell more equipment, government officials to
build their empires, and security academics to get grants. The upshot of all this
is that most of the damage done by terrorists to democratic countries comes
from the overreaction. Fortunately, electorates figure this out over time. In
Britain, where the IRA bombed us intermittently for a generation, the public
reaction to the 7/7 bombings was mostly a shrug.
Security engineers have to understand all this; we need to be able to put risks
and threats in context, make realistic assessments of what might go wrong, and
give our clients good advice. That depends on a wide understanding of what
has gone wrong over time with various systems; what sort of attacks have
worked, what their consequences were, and how they were stopped (if it was
worthwhile to do so). This book is full of case histories. I’ll talk about terrorism

5

6

Chapter 1

■

What Is Security Engineering?

specifically in Part III. For now, in order to set the scene, I’ll give a few brief
examples here of interesting security systems and what they are designed to
prevent.

1.3

Example 1 — A Bank

Banks operate a surprisingly large range of security-critical computer systems.
1. The core of a bank’s operations is usually a branch bookkeeping system.
This keeps customer account master files plus a number of journals that
record the day’s transactions. The main threat to this system is the bank’s
own staff; about one percent of bankers are fired each year, mostly for
petty dishonesty (the average theft is only a few thousand dollars). The
main defense comes from bookkeeping procedures that have evolved
over centuries. For example, each debit against one account must be
matched by an equal and opposite credit against another; so money can
only be moved within a bank, never created or destroyed. In addition,
large transfers of money might need two or three people to authorize
them. There are also alarm systems that look for unusual volumes or
patterns of transactions, and staff are required to take regular vacations
during which they have no access to the bank’s premises or systems.
2. One public face of the bank is its automatic teller machines. Authenticating transactions based on a customer’s card and personal identification
number — in such a way as to defend against both outside and inside
attack — is harder than it looks! There have been many epidemics of
‘phantom withdrawals’ in various countries when local villains (or bank
staff) have found and exploited loopholes in the system. Automatic teller
machines are also interesting as they were the first large scale commercial use of cryptography, and they helped establish a number of crypto
standards.
3. Another public face is the bank’s website. Many customers now do more
of their routine business, such as bill payments and transfers between
savings and checking accounts, online rather than at a branch. Bank
websites have come under heavy attack recently from phishing — from
bogus websites into which customers are invited to enter their passwords. The ‘standard’ internet security mechanisms designed in the
1990s, such as SSL/TLS, turned out to be ineffective once capable motivated opponents started attacking the customers rather than the bank.
Phishing is a fascinating security engineering problem mixing elements
from authentication, usability, psychology, operations and economics.
I’ll discuss it in detail in the next chapter.

1.4 Example 2 — A Military Base

4. Behind the scenes are a number of high-value messaging systems. These
are used to move large sums of money (whether between local banks
or between banks internationally); to trade in securities; to issue letters
of credit and guarantees; and so on. An attack on such a system is the
dream of the sophisticated white-collar criminal. The defense is a mixture of bookkeeping procedures, access controls, and cryptography.
5. The bank’s branches will often appear to be large, solid and prosperous,
giving customers the psychological message that their money is safe.
This is theatre rather than reality: the stone facade gives no real protection. If you walk in with a gun, the tellers will give you all the cash
you can see; and if you break in at night, you can cut into the safe or
strongroom in a couple of minutes with an abrasive wheel. The effective
controls these days center on the alarm systems — which are in constant
communication with a security company’s control center. Cryptography
is used to prevent a robber or burglar manipulating the communications and making the alarm appear to say ‘all’s well’ when it isn’t.
I’ll look at these applications in later chapters. Banking computer security is
important: until quite recently, banks were the main non-military market for
many computer security products, so they had a disproportionate influence
on security standards. Secondly, even where their technology isn’t blessed by
an international standard, it is often widely used in other sectors anyway.

1.4

Example 2 — A Military Base

Military systems have also been an important technology driver. They have
motivated much of the academic research that governments have funded into
computer security in the last 20 years. As with banking, there is not one single
application but many.
1. Some of the most sophisticated installations are the electronic warfare
systems whose goals include trying to jam enemy radars while preventing the enemy from jamming yours. This area of information warfare
is particularly instructive because for decades, well-funded research
labs have been developing sophisticated countermeasures, countercountermeasures and so on — with a depth, subtlety and range of deception strategies that are still not found elsewhere. As I write, in 2007, a lot
of work is being done on adapting jammers to disable improvised explosive devices that make life hazardous for allied troops in Iraq. Electronic
warfare has given many valuable insights: issues such as spoofing and
service-denial attacks were live there long before bankers and bookmakers started having problems with bad guys targeting their websites.

7

8

Chapter 1

■

What Is Security Engineering?

2. Military communication systems have some interesting requirements.
It is often not sufficient to just encipher messages: the enemy, on seeing traffic encrypted with somebody else’s keys, may simply locate the
transmitter and attack it. Low-probability-of-intercept (LPI) radio links are
one answer; they use a number of tricks that are now being adopted in
applications such as copyright marking. Covert communications are also
important in some privacy applications, such as in defeating the Internet
censorship imposed by repressive regimes.
3. Military organizations have some of the biggest systems for logistics and
inventory management, which differ from commercial systems in having
a number of special assurance requirements. For example, one may have
a separate stores management system at each different security level: a
general system for things like jet fuel and boot polish, plus a second
secret system for stores and equipment whose location might give away
tactical intentions. (This is very like the businessman who keeps separate
sets of books for his partners and for the tax man, and can cause similar
problems for the poor auditor.) There may also be intelligence systems
and command systems with even higher protection requirements. The
general rule is that sensitive information may not flow down to less
restrictive classifications. So you can copy a file from a Secret stores
system to a Top Secret command system, but not vice versa. The same
rule applies to intelligence systems which collect data using wiretaps:
information must flow up to the intelligence analyst from the target of
investigation, but the target must not know which of his communications
have been intercepted. Managing multiple systems with information
flow restrictions is a hard problem and has inspired a lot of research.
Since 9/11, for example, the drive to link up intelligence systems has
led people to invent search engines that can index material at multiple
levels and show users only the answers they are cleared to know.
4. The particular problems of protecting nuclear weapons have given rise
over the last two generations to a lot of interesting security technology,
ranging from electronic authentication systems that prevent weapons
being used without the permission of the national command authority, through seals and alarm systems, to methods of identifying people
with a high degree of certainty using biometrics such as iris patterns.
The civilian security engineer can learn a lot from all this. For example, many
early systems for inserting copyright marks into digital audio and video, which
used ideas from spread-spectrum radio, were vulnerable to desynchronisation
attacks that are also a problem for some spread-spectrum systems. Another
example comes from munitions management. There, a typical system enforces
rules such as ‘Don’t put explosives and detonators in the same truck’. Such

1.5 Example 3 — A Hospital

techniques can be recycled in food logistics — where hygiene rules forbid raw
and cooked meats being handled together.

1.5

Example 3 — A Hospital

From soldiers and food hygiene we move on to healthcare. Hospitals have a
number of interesting protection requirements — mostly to do with patient
safety and privacy.
1. Patient record systems should not let all the staff see every patient’s
record, or privacy violations can be expected. They need to implement
rules such as ‘nurses can see the records of any patient who has been
cared for in their department at any time during the previous 90 days’.
This can be hard to do with traditional computer security mechanisms
as roles can change (nurses move from one department to another) and
there are cross-system dependencies (if the patient records system ends
up relying on the personnel system for access control decisions, then the
personnel system may just have become critical for safety, for privacy or
for both).
2. Patient records are often anonymized for use in research, but this is
hard to do well. Simply encrypting patient names is usually not enough
as an enquiry such as ‘show me all records of 59 year old males who
were treated for a broken collarbone on September 15th 1966’ would
usually be enough to find the record of a politician who was known
to have sustained such an injury at college. But if records cannot be
anonymized properly, then much stricter rules have to be followed
when handling the data, and this increases the cost of medical research.
3. Web-based technologies present interesting new assurance problems
in healthcare. For example, as reference books — such as directories
of drugs — move online, doctors need assurance that life-critical data,
such as the figures for dosage per body weight, are exactly as published
by the relevant authority, and have not been mangled in some way.
Another example is that as doctors start to access patients’ records from
home or from laptops or even PDAs during house calls, suitable electronic authentication and encryption tools are starting to be required.
4. New technology can introduce risks that are just not understood. Hospital administrators understand the need for backup procedures to deal
with outages of power, telephone service and so on; but medical practice is rapidly coming to depend on the net in ways that are often not
documented. For example, hospitals in Britain are starting to use online
radiology systems: X-rays no longer travel from the X-ray machine to the

9

10

Chapter 1

■

What Is Security Engineering?

operating theatre in an envelope, but via a server in a distant town. So a
network failure can stop doctors operating just as much as a power failure. All of a sudden, the Internet turns into a safety-critical system, and
denial-of-service attacks might kill people.
We will look at medical system security too in more detail later. This is a
much younger field than banking IT or military systems, but as healthcare
accounts for a larger proportion of GNP than either of them in all developed
countries, and as hospitals are adopting IT at an increasing rate, it looks set to
become important. In the USA in particular, the HIPAA legislation — which
sets minimum standards for privacy — has made the sector a major client of
the information security industry.

1.6

Example 4 — The Home

You might not think that the typical family operates any secure systems. But
consider the following.
1. Many families use some of the systems we’ve already described. You
may use a web-based electronic banking system to pay bills, and in a few
years you may have encrypted online access to your medical records.
Your burglar alarm may send an encrypted ‘all’s well’ signal to the security company every few minutes, rather than waking up the neighborhood when something happens.
2. Your car probably has an electronic immobilizer that sends an encrypted
challenge to a radio transponder in the key fob; the transponder has to
respond correctly before the car will start. This makes theft harder and
cuts your insurance premiums. But it also increases the number of car
thefts from homes, where the house is burgled to get the car keys. The
really hard edge is a surge in car-jackings: criminals who want a getaway
car may just take one at gunpoint.
3. Early mobile phones were easy for villains to ‘clone’: users could
suddenly find their bills inflated by hundreds or even thousands of
dollars. The current GSM digital mobile phones authenticate themselves to the network by a cryptographic challenge-response protocol
similar to the ones used in car door locks and immobilizers.
4. Satellite TV set-top boxes decipher movies so long as you keep paying
your subscription. DVD players use copy control mechanisms based on
cryptography and copyright marking to make it harder to copy disks (or
to play them outside a certain geographic area). Authentication protocols can now also be used to set up secure communications on home networks (including WiFi, Bluetooth and HomePlug).

1.7 Definitions

5. In many countries, households who can’t get credit can get prepayment
meters for electricity and gas, which they top up using a smartcard or
other electronic key which they refill at a local store. Many universities use similar technologies to get students to pay for photocopier use,
washing machines and even soft drinks.
6. Above all, the home provides a haven of physical security and seclusion. Technological progress will impact this in many ways. Advances
in locksmithing mean that most common house locks can be defeated
easily; does this matter? Research suggests that burglars aren’t worried by locks as much as by occupants, so perhaps it doesn’t matter
much — but then maybe alarms will become more important for keeping intruders at bay when no-one’s at home. Electronic intrusion might
over time become a bigger issue, as more and more devices start to communicate with central services. The security of your home may come
to depend on remote systems over which you have little control.
So you probably already use many systems that are designed to enforce
some protection policy or other using largely electronic mechanisms. Over the
next few decades, the number of such systems is going to increase rapidly. On
past experience, many of them will be badly designed. The necessary skills are
just not spread widely enough.
The aim of this book is to enable you to design such systems better. To do
this, an engineer or programmer needs to learn about what systems there are,
how they work, and — at least as important — how they have failed in the
past. Civil engineers learn far more from the one bridge that falls down than
from the hundred that stay up; exactly the same holds in security engineering.

1.7

Definitions

Many of the terms used in security engineering are straightforward, but some
are misleading or even controversial. There are more detailed definitions of
technical terms in the relevant chapters, which you can find using the index.
In this section, I’ll try to point out where the main problems lie.
The first thing we need to clarify is what we mean by system. In practice,
this can denote:
1. a product or component, such as a cryptographic protocol, a smartcard
or the hardware of a PC;
2. a collection of the above plus an operating system, communications and
other things that go to make up an organization’s infrastructure;
3. the above plus one or more applications (media player, browser, word
processor, accounts / payroll package, and so on);

11

12

Chapter 1

■

What Is Security Engineering?

4. any or all of the above plus IT staff;
5. any or all of the above plus internal users and management;
6. any or all of the above plus customers and other external users.
Confusion between the above definitions is a fertile source of errors and
vulnerabilities. Broadly speaking, the vendor and evaluator communities focus
on the first (and occasionally) the second of them, while a business will focus on
the sixth (and occasionally the fifth). We will come across many examples of
systems that were advertised or even certified as secure because the hardware
was, but that broke badly when a particular application was run, or when
the equipment was used in a way the designers didn’t anticipate. Ignoring the
human components, and thus neglecting usability issues, is one of the largest
causes of security failure. So we will generally use definition 6; when we take
a more restrictive view, it should be clear from the context.
The next set of problems comes from lack of clarity about who the players are
and what they are trying to prove. In the literature on security and cryptology,
it’s a convention that principals in security protocols are identified by names
chosen with (usually) successive initial letters — much like hurricanes — and
so we see lots of statements such as ‘Alice authenticates herself to Bob’. This
makes things much more readable, but often at the expense of precision. Do we
mean that Alice proves to Bob that her name actually is Alice, or that she proves
she’s got a particular credential? Do we mean that the authentication is done
by Alice the human being, or by a smartcard or software tool acting as Alice’s
agent? In that case, are we sure it’s Alice, and not perhaps Cherie to whom
Alice lent her card, or David who stole her card, or Eve who hacked her PC?
By a subject I will mean a physical person (human, ET, . . .), in any role
including that of an operator, principal or victim. By a person, I will mean
either a physical person or a legal person such as a company or government1 .
A principal is an entity that participates in a security system. This entity can
be a subject, a person, a role, or a piece of equipment such as a PC, smartcard, or
card reader terminal. A principal can also be a communications channel (which
might be a port number, or a crypto key, depending on the circumstance). A
principal can also be a compound of other principals; examples are a group
(Alice or Bob), a conjunction (Alice and Bob acting together), a compound
role (Alice acting as Bob’s manager) and a delegation (Bob acting for Alice in
her absence). Beware that groups and roles are not the same. By a group I will
mean a set of principals, while a role is a set of functions assumed by different
persons in succession (such as ‘the officer of the watch on the USS Nimitz’
or ‘the president for the time being of the Icelandic Medical Association’). A
principal may be considered at more than one level of abstraction: e.g. ‘Bob
1
That some persons are not people may seem slightly confusing but it’s well established: blame
the lawyers.

1.7 Definitions

acting for Alice in her absence’ might mean ‘Bob’s smartcard representing Bob
who is acting for Alice in her absence’ or even ‘Bob operating Alice’s smartcard
in her absence’. When we have to consider more detail, I’ll be more specific.
The meaning of the word identity is controversial. When we have to be careful, I will use it to mean a correspondence between the names of two principals
signifying that they refer to the same person or equipment. For example, it
may be important to know that the Bob in ‘Alice acting as Bob’s manager’ is
the same as the Bob in ‘Bob acting as Charlie’s manager’ and in ‘Bob as branch
manager signing a bank draft jointly with David’. Often, identity is abused to
mean simply ‘name’, an abuse entrenched by such phrases as ‘user identity’
and ‘citizen’s identity card’. Where there is no possibility of being ambiguous,
I’ll sometimes lapse into this vernacular usage in order to avoid pomposity.
The definitions of trust and trustworthy are often confused. The following
example illustrates the difference: if an NSA employee is observed in a toilet
stall at Baltimore Washington International airport selling key material to a
Chinese diplomat, then (assuming his operation was not authorized) we can
describe him as ‘trusted but not trustworthy’. Hereafter, we’ll use the NSA
definition that a trusted system or component is one whose failure can break the
security policy, while a trustworthy system or component is one that won’t fail.
Beware, though, that there are many alternative definitions of trust. A UK
military view stresses auditability and fail-secure properties: a trusted systems
element is one ‘whose integrity cannot be assured by external observation of
its behaviour whilst in operation’. Other definitions often have to do with
whether a particular system is approved by authority: a trusted system might
be ‘a system which won’t get me fired if it gets hacked on my watch’ or even
‘a system which we can insure’. I won’t use either of these definitions. When
we mean a system which isn’t failure-evident, or an approved system, or an
insured system, I’ll say so.
The definition of confidentiality versus privacy versus secrecy opens another
can of worms. These terms clearly overlap, but equally clearly are not exactly
the same. If my neighbor cuts down some ivy at our common fence with the
result that his kids can look into my garden and tease my dogs, it’s not my
confidentiality that has been invaded. And the duty to keep quiet about the
affairs of a former employer is a duty of confidence, not of privacy.
The way I’ll use these words is as follows.
Secrecy is a technical term which refers to the effect of the mechanisms
used to limit the number of principals who can access information, such
as cryptography or computer access controls.
Confidentiality involves an obligation to protect some other person’s or
organization’s secrets if you know them.
Privacy is the ability and/or right to protect your personal information
and extends to the ability and/or right to prevent invasions of your

13

14

Chapter 1

■

What Is Security Engineering?

personal space (the exact definition of which varies quite sharply from
one country to another). Privacy can extend to families but not to legal
persons such as corporations.
For example, hospital patients have a right to privacy, and in order to
uphold this right the doctors, nurses and other staff have a duty of confidence
towards their patients. The hospital has no right of privacy in respect of its
business dealings but those employees who are privy to them may have a
duty of confidence. In short, privacy is secrecy for the benefit of the individual
while confidentiality is secrecy for the benefit of the organization.
There is a further complexity in that it’s often not sufficient to protect data,
such as the contents of messages; we also have to protect metadata, such as
logs of who spoke to whom. For example, many countries have laws making
the treatment of sexually transmitted diseases secret, and yet if a private eye
could find out that you were exchanging encrypted messages with an STD
clinic, he might well draw the conclusion that you were being treated there.
(A famous model in Britain recently won a privacy lawsuit against a tabloid
newspaper which printed a photograph of her leaving a meeting of Narcotics
Anonymous.) So anonymity can be just as important a factor in privacy (or
confidentiality) as secrecy. To make things even more complex, some writers
refer to what we’ve called secrecy as message content confidentiality and to
what we’ve called anonymity as message source (or destination) confidentiality.
In general, anonymity is hard. It’s difficult to be anonymous on your own;
you usually need a crowd to hide in. Also, our legal codes are not designed
to support anonymity: it’s much easier for the police to get itemized billing
information from the phone company, which tells them who called whom,
than it is to get an actual wiretap. (And it’s often very useful.)
The meanings of authenticity and integrity can also vary subtly. In the
academic literature on security protocols, authenticity means integrity plus
freshness: you have established that you are speaking to a genuine principal,
not a replay of previous messages. We have a similar idea in banking protocols.
In a country whose banking laws state that checks are no longer valid after
six months, a seven month old uncashed check has integrity (assuming it’s
not been altered) but is no longer valid. The military usage tends to be that
authenticity applies to the identity of principals and orders they give, while
integrity applies to stored data. Thus we can talk about the integrity of a
database of electronic warfare threats (it’s not been corrupted, whether by the
other side or by Murphy) but the authenticity of a general’s orders (which has
an overlap with the academic usage). However, there are some strange usages.
For example, one can talk about an authentic copy of a deceptive order given by
the other side’s electronic warfare people; here the authenticity refers to the act
of copying and storage. Similarly, a police crime scene officer will talk about
preserving the integrity of a forged check, by placing it in an evidence bag.

1.8 Summary

The last matter I’ll clarify here is the terminology which describes what we’re
trying to achieve. A vulnerability is a property of a system or its environment
which, in conjunction with an internal or external threat, can lead to a security
failure, which is a breach of the system’s security policy. By security policy I will
mean a succinct statement of a system’s protection strategy (for example, ‘each
credit must be matched by an equal and opposite debit, and all transactions
over $1,000 must be authorized by two managers’). A security target is a more
detailed specification which sets out the means by which a security policy will
be implemented in a particular product — encryption and digital signature
mechanisms, access controls, audit logs and so on — and which will be used as
the yardstick to evaluate whether the designers and implementers have done
a proper job. Between these two levels you may find a protection profile which
is like a security target except written in a sufficiently device-independent
way to allow comparative evaluations among different products and different
versions of the same product. I’ll elaborate on security policies, security targets
and protection profiles in later chapters. In general, the word protection will
mean a property such as confidentiality or integrity, defined in a sufficiently
abstract way for us to reason about it in the context of general systems rather
than specific implementations.

1.8

Summary

There is a lot of terminological confusion in security engineering, much
of which is due to the element of conflict. ‘Security’ is a terribly overloaded
word, which often means quite incompatible things to different people.
To a corporation, it might mean the ability to monitor all employees’ email
and web browsing; to the employees, it might mean being able to use email and
the web without being monitored. As time goes on, and security mechanisms
are used more and more by the people who control a system’s design to gain
some commercial advantage over the other people who use it, we can expect
conflicts, confusion and the deceptive use of language to increase.
One is reminded of a passage from Lewis Carroll:
‘‘When I use a word,’’ Humpty Dumpty said, in a rather scornful tone, ‘‘it
means just what I choose it to mean — neither more nor less.’’ ‘‘The question is,’’
said Alice, ‘‘whether you can make words mean so many different things.’’ ‘‘The
question is,’’ said Humpty Dumpty, ‘‘which is to be master — that’s all.’’
The security engineer should develop sensitivity to the different nuances of
meaning that common words acquire in different applications, and to be able to
formalize what the security policy and target actually are. That may sometimes
be inconvenient for clients who wish to get away with something, but, in general, robust security design requires that the protection goals are made explicit.

15

CHAPTER

2
Usability and Psychology
Humans are incapable of securely storing high-quality
cryptographic keys, and they have unacceptable speed and accuracy
when performing cryptographic operations. (They are also large,
expensive to maintain, difficult to manage, and they pollute the
environment. It is astonishing that these devices continue to be
manufactured and deployed. But they are sufficiently pervasive that
we must design our protocols around their limitations.)
— Kaufmann, Perlman and Speciner [698]
Only amateurs attack machines; professionals target people.
— Bruce Schneier

2.1

Introduction

Many real attacks exploit psychology at least as much as technology. The
fastest-growing online crime is phishing, in which victims are lured by an email
to log on to a website that appears genuine but that’s actually designed to
steal their passwords. Online frauds like phishing are often easier to do, and
harder to stop, than similar real-world frauds because most online protection
mechanisms are not anything like as intuitively usable or as difficult to forge
convincingly as their real-world equivalents; it is much easier for crooks to
build a bogus bank website that passes casual inspection than it is for them
to create a bogus bank in a shopping mall.
We’ve evolved social and psychological tools over millions of years to help
us deal with deception in face-to-face contexts, but these are little use to us
when we’re presented with an email that asks us to do something. It seems to be
harder to create useful asymmetry in usability, by which I mean that good use is
17

18

Chapter 2

■

Usability and Psychology

easier than bad use. We have some examples of asymmetry in physical objects:
a potato peeler is easier to use for peeling potatoes than a knife is, but a lot
harder to use for murder. However, much of the asymmetry on which we rely
in our daily business doesn’t just depend on formal exchanges — which can
be automated easily — but on some combination of physical objects, judgment
of people, and the supporting social protocols. (I’ll discuss this further in the
Introduction to Chapter 3.) So, as our relationships with employers, banks
and government become more formalised via online communication, and we
lose both physical and human context, the forgery of these communications
becomes more of a risk.
Deception, of various kinds, is now the greatest threat to online security. It
can be used to get passwords, or to compromise confidential information or
manipulate financial transactions directly. The most common way for private
investigators to steal personal information is pretexting — phoning someone
who has the information under a false pretext, usually by pretending to be
someone authorised to be told it. Such attacks are sometimes known collectively as social engineering. There are many other flavours. The quote from
Bruce Schneier at the head of this chapter appeared in a report of a stock
scam, where a bogus press release said that a company’s CEO had resigned
and its earnings would be restated. Several wire services passed this on, and
the stock dropped 61% until the hoax was exposed [1128]. Hoaxes and frauds
have always happened, but the Internet makes some of them easier, and lets
others be repackaged in ways that may bypass our existing controls (be they
personal intuitions, company procedures or even laws). We will be playing
catch-up for some time.
Another driver for the surge in attacks based on social engineering is
that people are getting better at technology. As designers learn how to
forestall the easier techie attacks, psychological manipulation of system users
or operators becomes ever more attractive. So the security engineer simply
must understand basic psychology and ‘security usability’, and one of the
biggest opportunities facing the research community is to learn more about
what works and why.

2.2

Attacks Based on Psychology

Hacking systems through the people who operate them may be growing
rapidly but is not new. Military and intelligence organisations have always
targeted each other’s staff; most of the intelligence successes of the old Soviet
Union were of this kind [77]. Private investigation agencies have not been far
behind. The classic attack of this type is pretexting.

2.2 Attacks Based on Psychology

2.2.1

Pretexting

Colleagues of mine did an experiment in England in 1996 to determine the
threat posed by pretexting to medical privacy. We trained the staff at a
health authority (a government-owned health insurer that purchased medical
services for a district of maybe 250,000 people) to identify and report falsepretext calls. A typical private eye would pretend to be a doctor involved in
the emergency care of a patient, and he could be detected because the phone
number he gave wasn’t that of the hospital at which he claimed to work. (The
story is told in detail later in the chapter on Multilateral Security.) We detected
about 30 false-pretext calls a week. Unfortunately, we were unable to persuade
the UK government to make this training mandatory for health authority staff.
Thirty attacks per week times 52 weeks in a year times 200 health authorities
in England is a lot of privacy compromise! Many countries have laws against
pretexting, including both the UK and the USA, yet there are people in both
countries who earn their living from it [411]. A typical case is reported in [449],
where a private eye collecting debts for General Motors was fined for conning
civil servants into giving out 250 people’s home addresses over the phone.
The year 2002 saw the publication of perhaps the most disturbing security
book ever, Kevin Mitnick’s ‘Art of Deception’. Mitnick, who got extensive
press coverage when he was arrested and convicted after breaking into phone
systems, related after his release from prison how almost all of his exploits
had involved social engineering. His typical hack was to pretend to a phone
company employee that he was a colleague, and solicit ‘help’ such as a
password. Ways of getting past a company’s switchboard and winning its
people’s trust have been taught for years in sales-training courses; Mitnick
became an expert at using them to defeat company security procedures, and
his book recounts a fascinating range of tricks [896].
Pretexting became world headline news in September 2006 when it emerged
that Hewlett-Packard chairwoman Patricia Dunn had hired private investigators who had used pretexting to obtain the phone records of other board
members of whom she was suspicious, and of journalists she considered hostile. She was forced to resign. The following month, the California Attorney
General filed felony charges and arrest warrants against her and three private
eyes. The charges were online crime, wire fraud, taking computer data and
using personal identifying information without authorization. In March 2007,
charges against her were dropped; a factor was that she was suffering from
cancer. Her codefendants pleaded no contest to lesser counts of fraudulent
wire communications, a misdemeanor, and got community service [93].
But fixing the problem is hard. Despite continuing publicity about pretexting,
there was an audit of the IRS in 2007 by the Treasury Inspector General for
Tax Administration, whose staff called 102 IRS employees at all levels, asked
for their user ids, and told them to change their passwords to a known value.

19

20

Chapter 2

■

Usability and Psychology

62 did so. Now nearly 100,000 IRS employees have access to tax return data,
so if you’re a US taxpayer there might be 60,000 people who might be fooled
into letting an intruder breach your financial privacy. What’s worse, this
happened despite similar audit tests in 2001 and 2004 [1131]. Now a number
of government departments, including Homeland Security, are planning to
launch phishing attacks on their own staff in order to gauge the effectiveness of
security education. In the UK, the privacy authorities announced a crackdown
and prosecuted a private detective agency that did blagging for top law
firms [779].
Resisting attempts by outsiders to inveigle your staff into revealing secrets
is known in military circles as operational security. Protecting really valuable secrets, such as unpublished financial data, not-yet-patented industrial
research or military plans, depends on limiting the number of people with
access, and also having strict doctrines about with whom they may be discussed and how. It’s not enough for rules to exist; you have to train all the staff
who have access to the confidential material, and explain to them the reasons
behind the rules. In our medical privacy example, we educated staff about
pretexting and trained them not to discuss medical records on the phone
unless they had initiated the call, and made it to a number they had got from
the phone book rather than from a caller. And once the staff have encountered,
detected and defeated a few pretexting attempts, they talk about it and the
message gets across loud and clear. Often the hardest people to educate are
the most senior; a consultancy sent the finance directors of 500 publicly-quoted
companies a USB memory stick as part of an anonymous invitation saying
‘For Your Chance to Attend the Party of a Lifetime’, and 46% of them put it
into their computers [701].
Intelligence-agency rules are very much tougher. Most of the operational
security effort goes into training staff in what not to do, instilling a culture
of discretion that shades well over into anonymity. And since foreign intelligence agencies make many fewer approaches to spooks than private eyes
make to medical-record clerks, a spymaster can’t rely on a robust detection
culture to spring up of its own accord. He has to have his own red team
constantly testing his staff to ensure that they take the paranoia business
seriously.
Some operational security measures are common sense, such as not tossing
out sensitive stuff in the trash. Less obvious is the need to train the people
you trust, even if they’re old friends. A leak of embarrassing emails that
appeared to come from the office of the UK Prime Minister and was initially
blamed on ‘hackers’ turned out to have been fished out of the trash at his
personal pollster’s home by a private detective called ‘Benji the Binman’ who
achieved instant celebrity status [828]. Governments have mostly adopted a
set of procedures whereby sensitive information is ‘classified’ and can only be
passed to people with an appropriate ‘clearance’, that is, background checks

2.2 Attacks Based on Psychology

and training. While this can become monstrously bureaucratic and wasteful,
it does still give a useful baseline for thinking about operational security, and
has led to the development of some protection technologies which I’ll discuss
later in the chapter on Multilevel Security. The disciplines used by banks to
prevent a rogue from talking a manager into sending him money are similar
in spirit but differ in detail; I discuss them in the chapter on Banking and
Bookkeeping.
Pretexting is mostly used for attacks on companies, but it’s starting to be
used more against individuals. Here’s the scam du jour in the USA, as I
write this in 2007: the bad guy phones you pretending to be a court official,
tells you you’ve been selected for jury duty, and demands your SSN and
date of birth. If you tell him, he applies for a credit card in your name. If
you tell him to get lost, he threatens you with arrest and imprisonment. Not
everyone has the self-confidence and legal knowledge to resist this kind of
sting.

2.2.2 Phishing
Phishing is in many ways a harder problem for a company to deal with
than pretexting, since (as with the last scam I mentioned) the targets are
not your staff but your customers. It is difficult enough to train the average
customer — and you can’t design simply for the average. If your security
systems are unusable by people who don’t speak English well, or who are
dyslexic, or who have learning difficulties, you are asking for serious legal
trouble, at least if you do business in civilised countries.
Phishing attacks against banks started in 2003, with half-a-dozen attempts
reported [299]. The early attacks imitated bank websites, but were both crude
and greedy; the attackers asked for all sorts of information such as ATM
PINs, and were also written in poor English. Most customers smelt a rat. The
attackers now use better psychology; they often reuse genuine bank emails,
with just the URLs changed, or send an email saying something like ‘Thank
you for adding a new email address to your PayPal account’ to provoke the
customer to log on to complain that they hadn’t. Of course, customers who use
the provided link rather than typing in www.paypal.com or using an existing
bookmark are likely to get their accounts emptied.
Losses are growing extremely rapidly (maybe $200 m in the USA in 2006,
£35 m / $70 m in the UK) although they are hard to tie down exactly as some
banks try to hold the customer liable and/or manipulate the accounting rules
to avoid reporting frauds. The phishing business has plenty room for growth.
Most UK losses in 2006 were sustained by one bank, while in the USA there are
perhaps half-a-dozen principal victims. We are only just starting to see largescale attacks on firms like eBay and Amazon, but I’m sure we will see many
more; when compromising a password lets you change the target’s email and

21

22

Chapter 2

■

Usability and Psychology

street addresses to your own, and then use their credit card to order a widescreen TV, the temptation is clear.
If you are a bank or an online retail business, then a number of factors
influence whether you get targeted. Some have to do with whether you’re
thought to be a wimp; banks that pursue fraudsters viciously and relentlessly
in the courts, well past the point of economic rationality, seem able to deter
attacks. The phishermen also prefer banks whose poor internal controls allow
large amounts of money to be moved abroad quickly, which lack effective
intrusion alarms, which take several days to check whether suspicious payments were authorised, and which don’t try too hard to retrieve those that
weren’t. (I will discuss internal controls later — see the chapter on Banking
and Bookkeeping.)
In the rest of this chapter, I’ll first visit some relevant basic psychology
and then apply it to the study of passwords — how you get users to choose
good passwords and enter them accurately, and what you can do to stop
users disclosing them to third parties. Finally there will be a brief section on
CAPTCHAs, the tests websites use to check that a user is a human rather than
a robot; these provide another angle on the differences between human minds
and software.

2.3

Insights from Psychology Research

I expect the interaction between security and psychology to be a big research
area over the next five years, just as security economics has been over the
last five. This is not just because of the growing number of attacks that target
users instead of (or as well as) technology. For example, terrorism is largely
about manipulating perceptions of risk; and even outside the national-security
context, many protection mechanisms are sold using scaremongering. (I’ll
return to the broader policy issues in Part III.)
Psychology is a huge subject, ranging from neuroscience through to clinical
topics, and spilling over into cognate disciplines from philosophy through
artificial intelligence to sociology. Although it has been studied for much
longer than computer science, our understanding of the mind is much less
complete: the brain is so much more complex. We still do not understand one
central problem — the nature of consciousness. We know that ‘the mind is
what the brain does’, yet the mechanisms that underlie our sense of self and
of personal history remain quite obscure.
Nonetheless a huge amount is known about the functioning of the mind
and the brain, and I expect we’ll get many valuable insights once we get
psychologists working together with security researchers on real problems.
In what follows I can only offer a helicopter tour of some ideas that appear
relevant to our trade.

2.3 Insights from Psychology Research

2.3.1 What the Brain Does Worse Than the Computer
Cognitive psychology deals with how we think, remember, make decisions
and even daydream. There are many well-known results; for example, it is
easier to memorise things that are repeated frequently, and it is easier to store
things in context. However, many of these insights are poorly understood by
systems developers. For example, most people have heard of George Miller’s
result that human short-term memory can cope with about seven (plus or
minus two) simultaneous choices [891] and, as a result, many designers limit
menu choices to about five. But this is not the right conclusion to draw. People
search for information first by recalling where to look, and then by scanning;
once you have found the relevant menu, scanning ten items is only twice as
hard as scanning five. The real limit on menu size is with spoken menus,
where the average user has difficulty dealing with more than three or four
choices [1039].
Our knowledge in this field has been significantly enhanced by the empirical know-how gained not just from lab experiments, but from the iterative
improvement of fielded systems. As a result, the centre of gravity has been
shifting from applied psychology to the human-computer interaction (HCI)
research community. HCI researchers not only model and measure human performance, including perception, motor control, memory and problem-solving;
they have also developed an understanding of how people’s mental models
of systems work, and of the techniques (such as task analysis and cognitive
walkthrough) that we can use to explore how people learn to use systems and
understand them.
Security researchers need to find ways of turning these ploughshares into
swords (the bad guys are already working on it). There are some obvious
low-hanging fruit; for example, the safety research community has done
a lot of work on characterising the errors people make when operating
equipment [1060]. It’s said that ‘to err is human’ and error research confirms
this: the predictable varieties of human error are rooted in the very nature
of cognition. The schemata, or mental models, that enable us to recognise
people, sounds and concepts so much better than computers do, also make us
vulnerable when the wrong model gets activated.
Human errors made while operating equipment fall into broadly three
categories, depending on where they occur in the ‘stack’: slips and lapses at
the level of skill, mistakes at the level of rules, and mistakes at the cognitive
level.
Actions performed often become a matter of skill, but this comes with
a downside: inattention can cause a practised action to be performed
instead of an intended one. We are all familiar with such capture errors;
an example is when you intend to go to the supermarket on the way

23

24

Chapter 2

■

Usability and Psychology

home from work but take the road home by mistake — as that’s what
you do most days. In computer systems, people are trained to click ‘OK’
to pop-up boxes as that’s often the only way to get the work done; some
attacks have used the fact that enough people will do this even when
they know they shouldn’t. (Thus Apple, unlike Microsoft, makes you
enter your password when installing software — as this is something
you do less often, you might just pause for thought.) Errors also commonly follow interruptions and perceptual confusion. One example
is the post-completion error: once they’ve accomplished their immediate
goal, people are easily distracted from tidying-up actions. More people
leave cards behind in ATMs that give them the money first and the card
back second.
Actions that people take by following rules are open to errors when they
follow the wrong rule. Various circumstances — such as information
overload — can cause people to follow the strongest rule they know, or
the most general rule, rather than the best one. Examples of phishermen
getting people to follow the wrong rule include using https (because
‘it’s secure’) and starting URLs with the impersonated bank’s name,
as www.citibank.secureauthentication.com — looking for the name
being for many people a stronger rule than parsing its position.
The third category of mistakes are those made by people for cognitive
reasons — they simply don’t understand the problem. For example,
Microsoft’s latest (IE7) anti-phishing toolbar is easily defeated by a
picture-in-picture attack, which I’ll describe later.
What makes security harder than safety is that we have a sentient attacker
who will try to provoke exploitable errors.
What can the defender do? Well, we expect the attacker to use errors whose
effect is predictable, such as capture errors. We also expect him to look for,
or subtly create, exploitable dissonances between users’ mental models of a
system and its actual logic. Given a better understanding of this, we might
try to engineer countermeasures — perhaps a form of cognitive walkthrough
aimed at identifying attack points, just as a code walkthough can be used to
search for software vulnerabilities.

2.3.2

Perceptual Bias and Behavioural Economics

Perhaps the most promising field of psychology for security folks to mine in
the short term is that which studies the heuristics that people use, and the
biases that influence them, when making decisions. This discipline, known
as behavioural economics or decision science, sits at the boundary of psychology
and economics. It examines the ways in which people’s decision processes
depart from the rational behaviour modeled by economists; Daniel Kahneman

2.3 Insights from Psychology Research

won the Nobel Prize in economics in 2002 for launching this field (along with
the late Amos Tversky). One of his insights was that the heuristics we use
in everyday judgement and decision making lie somewhere between rational
thought and the unmediated input from the senses [679].
Kahneman and Tversky did extensive experimental work on how people
made decisions faced with uncertainty. They developed prospect theory which
models risk aversion, among other things: in many circumstances, people
dislike losing $100 they already have more than they value winning $100.
That’s why marketers talk in terms of ‘discount’ and ‘saving’ — by framing an
action as a gain rather than as a loss makes people more likely to take it. We’re
also bad at calculating probabilities, and use all sorts of heuristics to help us
make decisions: we base inferences on familiar or easily-imagined analogies
(the availability heuristic whereby easily-remembered data have more weight in
mental processing), and by comparison with recent experiences (the anchoring
effect whereby we base a judgement on an initial guess or comparison and then
adjust it if need be). We also worry too much about unlikely events.
The channels through which we experience things also matter (we’re more
likely to be sceptical about things we’ve heard than about things we’ve seen).
Another factor is that we evolved in small social groups, and the behaviour
appropriate here isn’t the same as in markets; indeed, many frauds work by
appealing to our atavistic instincts to trust people more in certain situations
or over certain types of decision. Other traditional vices now studied by
behavioural economists range from our tendency to procrastinate to our
imperfect self-control.
This tradition is not just relevant to working out how likely people are to
click on links in phishing emails, but to the much deeper problem of the public
perception of risk. Many people perceive terrorism to be a much worse threat
than food poisoning or road traffic accidents: this is irrational, but hardly
surprising to a behavioural economist, as we overestimate the small risk of
dying in a terrorist attack not just because it’s small but because of the visual
effect of the 9/11 TV coverage and the ease of remembering the event. (There
are further factors, which I’ll discuss in Chapter 24 when we discuss terrorism.)
The misperception of risk underlies many other public-policy problems.
The psychologist Daniel Gilbert, in an article provocatively entitled ‘If only
gay sex caused global warming’, discusses why we are much more afraid of
terrorism than of climate change. First, we evolved to be much more wary of
hostile intent than of nature; 100,000 years ago, a man with a club (or a hungry
lion) was a much worse threat than a thunderstorm. Second, global warming
doesn’t violate anyone’s moral sensibilities; third, it’s a long-term threat rather
than a clear and present danger; and fourth, we’re sensitive to rapid changes
in the environment rather than slow ones [526].
Bruce Schneier lists more biases: we are less afraid when we’re in control,
such as when driving a car, as opposed to being a passenger in a car or

25

26

Chapter 2

■

Usability and Psychology

airplane; we are more afraid of risks to which we’ve been sensitised, for
example by gruesome news coverage; and we are more afraid of uncertainty,
that is, when the magnitude of the risk is unknown (even when it’s small). And
a lot is known on the specific mistakes we’re likely to make when working out
probabilities and doing mental accounting [1129, 1133].
Most of us are not just more afraid of losing something we have, than of
not making a gain of equivalent value, as prospect theory models. We’re also
risk-averse in that most people opt for a bird in the hand rather than two in
the bush. This is thought to be an aspect of satisficing — as situations are often
too hard to assess accurately, we have a tendency to plump for the alternative
that’s ‘good enough’ rather than face the cognitive strain of trying to work out
the odds perfectly, especially when faced with a small transaction. Another
aspect of this is that many people just plump for the standard configuration
of a system, as they assume it will be good enough. This is one reason why
secure defaults matter1 .
There is a vast amount of material here that can be exploited by the
fraudster and the terrorist, as well as by politicians and other marketers. And
as behavioural psychology gets better understood, the practice of marketing
gets sharper too, and the fraudsters are never far behind. And the costs to
business come not just from crime directly, but even more from the fear of
crime. For example, many people don’t use electronic banking because of a
fear of fraud that is exaggerated (at least in the USA with its tough consumerprotection laws): so banks pay a fortune for the time of branch and call-center
staff. So it’s not enough for the security engineer to stop bad things happening;
you also have to reassure people. The appearance of protection can matter just
as much as the reality.

2.3.3

Different Aspects of Mental Processing

Many psychologists see the mind as composed of interacting rational and
emotional components — ‘heart’ and ‘head’, or ‘affective’ and ‘cognitive’ systems. Studies of developmental biology have shown that, from an early age,
we have different mental processing systems for social phenomena (such as
recognising parents and siblings) and physical phenomena. Paul Bloom has
written a provocative book arguing that the tension between them explains
why many people are natural dualists — that is, they believe that mind and
body are basically different [194]. Children try to explain what they see using
their understanding of physics, but when this falls short, they explain phenomena in terms of deliberate action. This tendency to look for affective
1
In fact, behavioral economics has fostered a streak of libertarian paternalism in the policy world
that aims at setting good defaults in many spheres. An example is the attempt to reduce poverty
in old age by making pension plans opt-out rather than opt-in.

2.3 Insights from Psychology Research

explanations in the absence of material ones has survival value to the young,
as it disposes them to get advice from parents or other adults about novel
natural phenomena. According to Bloom, it has a significant side-effect: it
predisposes humans to believe that body and soul are different, and thus lays
the ground for religious belief. This argument may not overwhelm the faithful
(who can retort that Bloom simply stumbled across a mechanism created by
the Intelligent Designer to cause us to have faith in Him). But it may have
relevance for the security engineer.
First, it goes some way to explaining the fundamental attribution error —
people often err by trying to explain things by intentionality rather than by
situation. Second, attempts to curb phishing by teaching users about the gory
design details of the Internet — for example, by telling them to parse URLs
in emails that seem to come from a bank — will be of limited value if users
get bewildered. If the emotional is programmed to take over whenever the
rational runs out, then engaging in a war of technical measures and countermeasures with the phishermen is fundamentally unsound. Safe defaults would
be better — such as ‘Our bank will never, ever send you email. Any email that
purports to come from us is fraudulent.’
It has spilled over recently into behavioural economics via the affect heuristic,
explored by Paul Slovic and colleagues [1189]. The idea is that by asking an
emotional question (such as ‘How many dates did you have last month?’)
you can get people to answer subsequent questions using their hearts more
than their minds, which can make people insensitive to probability. This
work starts to give us a handle on issues from people’s risky behaviour with
porn websites to the use of celebrities in marketing (and indeed in malware).
Cognitive overload also increases reliance on affect: so a bank that builds a
busy website may be able to sell more life insurance, but it’s also likely to
make its customers more vulnerable to phishing. In the other direction, events
that evoke a feeling of dread — from cancer to terrorism — scare people more
than the naked probabilities justify.
Our tendency to explain things by intent rather than by situation is reinforced
by a tendency to frame decisions in social contexts; for example, we’re more
likely to trust people against whom we can take vengeance. (I’ll discuss
evolutionary game theory, which underlies this, in the chapter on Economics.)

2.3.4

Differences Between People

Most information systems are designed by men, and yet over half their
users may be women. Recently people have realised that software can create
barriers to females, and this has led to research work on ‘gender HCI’ — on
how software should be designed so that women as well as men can use
it effectively. For example, it’s known that women navigate differently from
men in the real world, using peripheral vision more, and it duly turns

27

28

Chapter 2

■

Usability and Psychology

out that larger displays reduce gender bias. Other work has focused on
female programmers, especially end-user programmers working with tools
like spreadsheets. It turns out that women tinker less than males, but more
effectively [139]. They appear to be more thoughtful, but lower self-esteem and
higher risk-aversion leads them to use fewer features. Given that many of the
world’s spreadsheet users are women, this work has significant implications
for product design.
No-one seems to have done any work on gender and security usability, yet
reviews of work on gender psychology (such as [1012]) suggest many points
of leverage. One formulation, by Simon Baron-Cohen, classifies human brains
into type S (systematizers) and type E (empathizers) [120]. Type S people
are better at geometry and some kinds of symbolic reasoning, while type
Es are better at language and multiprocessing. Most men are type S, while
most women are type E, a relationship that Baron-Cohen believes is due to
fetal testosterone levels. Of course, innate abilities can be modulated by many
developmental and social factors. Yet, even at a casual reading, this material
makes me suspect that many security mechanisms are far from gender-neutral.
Is it unlawful sex discrimination for a bank to expect its customers to detect
phishing attacks by parsing URLs?

2.3.5

Social Psychology

This discipline attempts to explain how the thoughts, feelings, and behaviour
of individuals are influenced by the actual, imagined, or implied presence of
others. It has many aspects, from the identity that people derive from belonging
to groups, through the self-esteem we get by comparing ourselves with others.
It may be particularly useful in understanding persuasion; after all, deception
is the twin brother of marketing. The growth of social-networking systems will
lead to peer pressure being used as a tool for deception, just as it is currently
used as a tool for marketing fashions.
Social psychology has been entangled with the security world longer than
many other parts of psychology through its relevance to propaganda, interrogation and aggression. Three particularly famous experiments in the 20th
century illuminated this. In 1951, Solomon Asch showed that people could
be induced to deny the evidence of their own eyes in order to conform to
a group. Subjects judged the lengths of lines after hearing wrong opinions
from other group members, who were actually the experimenter’s associates.
Most subjects gave in and conformed, with only 29% resisting the bogus
majority [90].
Stanley Milgram was inspired by the 1961 trial of Adolf Eichmann to
investigate how many experimental subjects were prepared to administer
severe electric shocks to an actor playing the role of a ‘learner’ at the behest
of an experimenter playing the role of the ‘teacher’ — even when the ‘learner’

2.3 Insights from Psychology Research

appeared to be in severe pain and begged the subject to stop. This experiment
was designed to measure what proportion of people will obey an authority
rather than their conscience. Most will — consistently over 60% of people will
do downright immoral things if they are told to [888].
The third of these was the Stanford Prisoner Experiment which showed that
normal people can behave wickedly even in the absence of orders. In 1971,
experimenter Philip Zimbardo set up a ‘prison’ at Stanford where 24 students
were assigned at random to the roles of 12 warders and 12 inmates. The aim
of the experiment was to discover whether prison abuses occurred because
warders (and possibly prisoners) were self-selecting. However, the students
playing the role of warders rapidly became sadistic authoritarians, and the
experiment was halted after six days on ethical grounds [1377].
Abuse of authority, whether real or ostensible, is a major issue for people
designing operational security measures. During the period 1995–2005, a
hoaxer calling himself ‘Officer Scott’ ordered the managers of over 68 US
stores and restaurants in 32 US states (including at least 17 McDonalds’ stores)
to detain some young employee on suspicion of theft and strip-search her or
him. Various other degradations were ordered, including beatings and sexual
assaults [1351]. A former prison guard was tried for impersonating a police
officer but acquitted. At least 13 people who obeyed the caller and did searches
were charged with crimes, and seven were convicted. MacDonald’s got sued
for not training its store managers properly, even years after the pattern of
hoax calls was established; and in October 2007, a jury ordered McDonalds
to pay $6.1 million dollars to Louise Ogborn, one of the victims, who had
been strip-searched when an 18-year-old employee. It was an unusually nasty
case, as the victim was then left by the store manager in the custody of
her boyfriend, who forced her to perform oral sex on him. The boyfriend
got five years, and the manager pleaded guilty to unlawfully detaining
Ogborn. When it came to the matter of damages, McDonalds argued that
Ogborn was responsible for whatever damages she suffered for not realizing
it was a hoax, and that the store manager had failed to apply common
sense. A Kentucky jury didn’t buy this and ordered McDonalds to pay up.
The store manager also sued, saying she too was the victim of McDonalds’
negligence to warn her of the hoax, and got $1.1 million [740]. So as of
2007, US employers seem to have a legal duty to train their staff to resist
pretexting.
But what about a firm’s customers? There is a lot of scope for phishermen
to simply order bank customers to reveal their security data. Bank staff
routinely tell their customers to do this, even when making unsolicited calls.
I’ve personally received an unsolicited call from my bank saying ‘Hello, this
is Lloyds TSB, can you tell me your mother’s maiden name?’ and caused the
caller much annoyance by telling her to get lost. Most people don’t, though.
ATM card thieves already called their victims in the 1980s and, impersonating

29

30

Chapter 2

■

Usability and Psychology

bank or police officers, have ordered them to reveal PINs ‘so that your card can
be deactivated’. The current scam — as of December 2007 — is that callers who
pretend to be from Visa say they are conducting a fraud investigation. After
some rigmarole they say that some transactions to your card were fraudulent,
so they’ll be issuing a credit. But they need to satisfy themselves that you are
still in possession of your card: so can you please read out the three security
digits on the signature strip? A prudent system designer will expect a lot more
of this, and will expect the courts to side with the customers eventually. If you
train your customers to do something that causes them to come to harm, you
can expect no other outcome.
Another interesting offshoot of social psychology is cognitive dissonance
theory. People are uncomfortable when they hold conflicting views; they
seek out information that confirms their existing views of the world and
of themselves, and try to reject information that conflicts with their views
or might undermine their self-esteem. One practical consequence is that
people are remarkably able to persist in wrong courses of action in the
face of mounting evidence that things have gone wrong [1241]. Admitting
to yourself or to others that you were duped can be painful; hustlers know
this and exploit it. A security professional should ‘feel the hustle’ — that
is, be alert for a situation in which recently established social cues and
expectations place you under pressure to ‘just do’ something about which
you’d normally have reservations, so that you can step back and ask yourself
whether you’re being had. But training people to perceive this is hard enough,
and getting the average person to break the social flow and say ‘stop!’ is
really hard.

2.3.6

What the Brain Does Better Than the Computer

Psychology isn’t all doom and gloom for our trade, though. There are tasks
that the human brain performs much better than a computer. We are extremely
good at recognising other humans visually, an ability shared by many primates.
We are good at image recognition generally; a task such as ‘pick out all scenes
in this movie where a girl rides a horse next to water’ is trivial for a human
child yet a hard research problem in image processing. We’re also better than
machines at understanding speech, particularly in noisy environments, and at
identifying speakers.
These abilities mean that it’s possible to devise tests that are easy for humans
to pass but hard for machines — the so-called ‘CAPTCHA’ tests that you often
come across when trying to set up an online account or posting to a bulletin
board. I will describe CAPTCHAs in more detail later in this chapter. They are
a useful first step towards introducing some asymmetry into the interactions
between people and machines, so as to make the bad guy’s job harder than the
legitimate user’s.

2.4 Passwords

2.4

Passwords

In this section, I will focus on the management of passwords as a simple,
important and instructive context in which usability, applied psychology and
security meet. Passwords are one of the biggest practical problems facing
security engineers today. In fact, as the usability researcher Angela Sasse puts
it, it’s hard to think of a worse authentication mechanism than passwords, given
what we know about human memory: people can’t remember infrequentlyused, frequently-changed, or many similar items; we can’t forget on demand;
recall is harder than recognition; and non-meaningful words are more difficult.
The use of passwords imposes real costs on business: the UK phone company
BT has a hundred people in its password-reset centre.
There are system and policy issues too: as people become principals in more
and more electronic systems, the same passwords get used over and over
again. Not only may attacks be carried out by outsiders guessing passwords,
but by insiders in other systems. People are now asked to choose passwords
for a large number of websites that they visit rarely. Does this impose an
unreasonable burden?
Passwords are not, of course, the only way of authenticating users to
systems. There are basically three options. The person may retain physical
control of the device — as with a remote car door key. The second is that
she presents something she knows, such as a password. The third is to use
something like a fingerprint or iris pattern, which I’ll discuss in the chapter
on Biometrics. (These options are commonly summed up as ‘something you
have, something you know, or something you are’ — or, as Simson Garfinkel
engagingly puts it, ‘something you had once, something you’ve forgotten, or
something you once were’.) But for reasons of cost, most systems take the
second option; and even where we use a physical token such as a one-time
password generator, it is common to use another password as well (whether
to lock it, or as an additional logon check) in case it gets stolen. Biometrics are
also commonly used in conjunction with passwords, as you can’t change your
fingerprint once the Mafia gets to know it. So, like it or not, passwords are the
(often shaky) foundation on which much of information security is built.
Some passwords have to be ‘harder’ than others, the principal reason being
that sometimes we can limit the number of guesses an opponent can make
and sometimes we cannot. With an ATM PIN, the bank can freeze the account
after three wrong guesses, so a four-digit number will do. But there are many
applications where it isn’t feasible to put a hard limit on the number of guesses,
such as where you encrypt a document with a password; someone who gets
hold of the ciphertext can try passwords till the cows come home. In such
applications, we have to try to get people to use longer passwords that are
really hard to guess.

31

32

Chapter 2

■

Usability and Psychology

In addition to things that are ‘obviously’ passwords, such as your computer
password and your bank card PIN, many other things (and combinations of
things) are used for the same purpose. The most notorious are social security
numbers, and your mother’s maiden name, which many organisations use to
recognize you. The ease with which such data can be guessed, or found out
from more or less public sources, has given rise to a huge industry of so-called
‘identity theft’ [458]. Criminals obtain credit cards, mobile phones and other
assets in your name, loot them, and leave you to sort out the mess. In the USA,
about half a million people are the ‘victims’ of this kind of fraud each year2 .
So passwords matter, and managing them is a serious real world problem
that mixes issues of psychology with technical issues. There are basically three
broad concerns, in ascending order of importance and difficulty:
1. Will the user enter the password correctly with a high enough
probability?
2. Will the user remember the password, or will they have to either write it
down or choose one that’s easy for the attacker to guess?
3. Will the user break the system security by disclosing the password
to a third party, whether accidentally, on purpose, or as a result of
deception?

2.4.1

Difficulties with Reliable Password Entry

Our first human-factors issue is that if a password is too long or complex,
users might have difficulty entering it correctly. If the operation they are
trying to perform is urgent, this might have safety implications. If customers
have difficulty entering software product activation codes, this can generate
expensive calls to your support desk.
One application in which this is important is encrypted access codes. By
quoting a reservation number, we get access to a hotel room, a rental car
or an airline ticket. Activation codes for software and other products are
often alphanumeric representations of encrypted data, which can be a 64-bit
or 128-bit string with symmetric ciphers and hundreds of bits when publickey cryptography is used. As the numbers get longer, what happens to the
error rate?
2
I write ‘identity theft’ in quotes as it’s a propaganda term for the old-fashioned offence of
impersonation. In the old days, if someone went to a bank, pretended to be me, borrowed money
from them and vanished, then that was the bank’s problem, not mine. In the USA and the UK,
banks have recently taken to claiming that it’s my identity that’s been stolen rather than their
money, and that this somehow makes me liable. So I also parenthesise ‘victims’ — the banks are
the real victims, except insofar as they commit secondary fraud against the customer. There’s an
excellent discussion of this by Adam Shostack and Paul Syverson in [1166].

2.4 Passwords

An interesting study was done in South Africa in the context of prepaid
electricity meters used to sell electricity in areas where the customers have no
credit rating and often not even an address. With the most common make of
meter, the customer hands some money to a sales agent, and in return gets
one or more 20-digit numbers printed out on a receipt. He takes this receipt
home and enters the numbers at a keypad in his meter. These numbers are
encrypted commands, whether to dispense electricity, to change the tariff or
whatever; the meter decrypts them and acts on them.
When this meter was introduced, its designers worried that since a third
of the population was illiterate, and since people might get lost halfway
through entering the number, the system might be unusable. But it turned
out that illiteracy was not a problem: even people who could not read had
no difficulty with numbers (‘everybody can use a phone’, as one of the
engineers said). Entry errors were a greater problem, but were solved by
printing the twenty digits in two rows, with three and two groups of four
digits respectively [59].
A quite different application is the firing codes for U.S. nuclear weapons.
These consist of only 12 decimal digits. If they are ever used, the operators
may be under extreme stress, and possibly using improvised or obsolete
communications channels. Experiments suggested that 12 digits was the
maximum that could be conveyed reliably in such circumstances.

2.4.2

Difficulties with Remembering the Password

Our second psychological issue with passwords is that people often find them
hard to remember [245, 1379]. Twelve to twenty digits may be fine when they
are simply copied from a telegram or a meter ticket, but when customers are
expected to memorize passwords, they either choose values which are easy for
attackers to guess, or write them down, or both. In fact, the password problem
has been neatly summed up as: ‘‘Choose a password you can’t remember, and
don’t write it down.’’
The problems are not limited to computer access. For example, one chain of
hotels in France introduced completely unattended service. You would turn
up at the hotel, swipe your credit card in the reception machine, and get a
receipt with a numerical access code which would unlock your room door. To
keep costs down, the rooms did not have en-suite bathrooms, so guests had to
use communal facilities. The usual failure mode was that a guest, having gone
to the bathroom, would forget his access code. Unless he had taken the receipt
with him, he’d end up having to sleep on the bathroom floor until the staff
arrived the following morning.
Problems related to password memorability can be discussed under four
main headings: naive password choice, user abilities and training, design
errors, and operational failures.

33

34

Chapter 2

2.4.3

■

Usability and Psychology

Naive Password Choice

Since at least the mid-1980s, people have studied what sort of passwords are
chosen by users who are left to their own devices. The results are depressing.
People will use spouses’ names, single letters, or even just hit carriage return
giving an empty string as their password. So some systems started to require
minimum password lengths, or even check user entered passwords against a
dictionary of bad choices. However, password quality enforcement is harder
than you might think. Fred Grampp and Robert Morris’s classic paper on
Unix security [550] reports that after software became available which forced
passwords to be at least six characters long and have at least one nonletter,
they made a file of the 20 most common female names, each followed by a
single digit. Of these 200 passwords, at least one was in use on each of several
dozen machines they examined.
A well-known study was conducted by Daniel Klein who gathered 25,000
Unix passwords in the form of encrypted password files and ran cracking
software to guess them [720]. He found that 21–25% of passwords could be
guessed depending on the amount of effort put in. Dictionary words accounted
for 7.4%, common names for 4%, combinations of user and account name 2.7%,
and so on down a list of less probable choices such as words from science
fiction (0.4%) and sports terms (0.2%). Some of these were straighforward
dictionary searches; others used patterns. For example, the algorithm for
constructing combinations of user and account names would take an account
‘klone’ belonging to the user ‘Daniel V. Klein’ and try passwords such as klone,
klone1, klone 123, dvk, dvkdvk, leinad, neilk, DvkkvD, and so on.
Many firms require users to change passwords regularly, but this tends
to backfire. According to one report, when users were compelled to change
their passwords and prevented from using the previous few choices, they
changed passwords rapidly to exhaust the history list and get back to their
favorite password. A response, of forbidding password changes until after
15 days, meant that users couldn’t change compromised passwords without
help from an administrator [1008]. A large healthcare organisation in England
is only now moving away from a monthly change policy; the predictable result
was a large number of password resets at month end (to cope with which,
sysadmins reset passwords to a well-known value). In my own experience,
insisting on alphanumeric passwords and also forcing a password change once
a month led people to choose passwords such as ‘julia03’ for March, ‘julia04’
for April, and so on.
So when our university’s auditors write in their annual report each year that
we should have a policy of monthly enforced password change, my response
is to ask the chair of our Audit Committee when we’ll get a new lot of auditors.
Even among the general population, there is some evidence that many people now choose slightly better passwords; passwords retrieved from phishing

2.4 Passwords

sites typically contain numbers as well as letters, while the average password
length has gone up from six to eight characters and the most common password is not ‘password’ but ‘password1’ [1130]. One possible explanation is that
many people try to use the same password everywhere, and the deployment
of password checking programs on some websites trains them to use longer
passwords with numbers as well as letters [302].

2.4.4 User Abilities and Training
Sometimes you really can train users. In a corporate or military environment
you can try to teach them to choose good passwords, or issue them with
random passwords, and insist that passwords are treated the same way as the
data they protect. So bank master passwords go in the vault overnight, while
military ‘Top Secret’ passwords must be sealed in an envelope, in a safe, in a
room that’s locked when not occupied, in a building patrolled by guards. You
can run background checks on everyone with access to any terminals where
the passwords can be used. You can encrypt passwords along with data in
transit between secure sites. You can send guards round at night to check
that no-one’s left a note of a password lying around. You can operate a clean
desk policy so that a password can’t be overlooked in a pile of papers on
a desk. You can send your guards round the building every night to clean all
desks every night.
Even if you’re running an e-commerce website, you are not completely
helpless: you can give your users negative feedback if they choose bad
passwords. For example, you might require that passwords be at least eight
characters long and contain at least one nonletter. But you will not want
to drive your customers away. And even in the Army, you do not want to
order your soldiers to do things they can’t; then reality and regulations will
drift apart, you won’t really know what’s going on, and discipline will be
undermined. So what can you realistically expect from users when it comes to
choosing and remembering passwords?
Colleagues and I studied the benefits that can be obtained by training
users [1365]. While writing the first edition of this book, I could not find any
account of experiments on this that would hold water by the standards of
applied psychology (i.e., randomized controlled trials with big enough groups
for the results to be statistically significant). The closest I found was a study
of the recall rates, forgetting rates, and guessing rates of various types of
password [245]; this didn’t tell us the actual (as opposed to likely) effects
of giving users various kinds of advice. We therefore selected three groups of
about a hundred volunteers from our first year science students.
The red (control) group was given the usual advice (password at least six
characters long, including one nonletter).

35

36

Chapter 2

■

Usability and Psychology

The green group was told to think of a passphrase and select letters from
it to build a password. So ‘It’s 12 noon and I am hungry’ would give
‘I’S12&IAH’.
The yellow group was told to select eight characters (alpha or numeric)
at random from a table we gave them, write them down, and destroy
this note after a week or two once they’d memorized the password.
What we expected to find was that the red group’s passwords would be
easier to guess than the green group’s which would in turn be easier than
the yellow group’s; and that the yellow group would have the most difficulty
remembering their passwords (or would be forced to reset them more often),
followed by green and then red. But that’s not what we found.
About 30% of the control group chose passwords that could be guessed
using cracking software (which I discuss later), versus about 10 percent for
the other two groups. So passphrases and random passwords seemed to be
about equally effective. When we looked at password reset rates, there was no
significant difference between the three groups. When we asked the students
whether they’d found their passwords hard to remember (or had written them
down), the yellow group had significantly more problems than the other two;
but there was no significant difference between red and green.
The conclusions we drew were as follows.
For users who follow instructions, passwords based on mnemonic
phrases offer the best of both worlds. They are as easy to remember as
naively selected passwords, and as hard to guess as random passwords.
The problem then becomes one of user compliance. A significant number
of users (perhaps a third of them) just don’t do what they’re told.
So, while a policy of centrally-assigned, randomly selected passwords may
work for the military, its value comes from the fact that the passwords are
centrally assigned (thus compelling user compliance) rather than from the fact
that they’re random (as mnemonic phrases would do just as well).
But centrally-assigned passwords are often inappropriate. When you are
offering a service to the public, your customers expect you to present broadly
the same interfaces as your competitors. So you must let users choose their own
website passwords, subject to some lightweight algorithm to reject passwords
that are too short or otherwise ‘clearly bad’. In the case of bank cards, users
expect a bank-issued initial PIN plus the ability to change the PIN afterwards
to one of their choosing (though again you may block a ‘clearly bad’ PIN such
as 0000 or 1234). There can also be policy reasons not to issue passwords:
for example, in Europe you can’t issue passwords for devices that generate
electronic signatures, as this could enable the system administrator to get at
the signing key and forge messages, which would destroy the evidential value
of the signature. By law, users must choose their own passwords.

2.4 Passwords

So the best compromise will often be a password checking program that
rejects ‘clearly bad’ user choices, plus a training program to get your compliant
users to choose mnemonic passwords. Password checking can be done using a
program like crack to filter user choices; other programs understand language
statistics and reject passwords that are too likely to be chosen by others at
random [353, 163]; another option is to mix the two ideas using a suitable
coding scheme [1207].

2.4.4.1

Design Errors

Attempts to make passwords memorable are a frequent source of severe design
errors — especially with the many systems built rapidly by unskilled people
in the dotcom rush by businesses to get online.
An important example of how not to do it is to ask for ‘your mother’s
maiden name’. A surprising number of banks, government departments and
other organisations authenticate their customers in this way. But there are two
rather obvious problems. First, your mother’s maiden name is easy for a thief
to find out, whether by asking around or using online genealogical databases.
Second, asking for a maiden name makes assumptions which don’t hold for
all cultures, so you can end up accused of discrimination: Icelanders have no
surnames, and women from many other countries don’t change their names
on marriage. Third, there is often no provision for changing ‘your mother’s
maiden name’, so if it ever becomes known to a thief your customer would
have to close bank accounts (and presumably reopen them elsewhere). And
even if changes are possible, and a cautious customer decides that from now on
her mother’s maiden name is going to be Yngstrom (or even ‘yGt5r4ad’) rather
than Smith, there are further problems. She might be worried about breaking
her credit card agreement, and perhaps invalidating her insurance cover, by
giving false data. So smart customers will avoid your business; famous ones,
whose mothers’ maiden names are in Who’s Who, should certainly shun you.
Finally, people are asked to give their mother’s maiden name to a lot of
organisations, any one of which might have a crooked employee. (You could
always try to tell ‘Yngstrom’ to your bank, ‘Jones’ to the phone company,
‘Geraghty’ to the travel agent, and so on; but data are shared extensively
between companies, so you could easily end up confusing their systems — not
to mention yourself).
Some organisations use contextual security information. My bank asks its
business customers the value of the last check from their account that was
cleared. In theory, this could be helpful: even if someone compromises my password — such as by overhearing me doing a transaction on the telephone — the
security of the system usually recovers more or less automatically. The details
bear some attention though. When this system was first introduced, I wondered about the risk that a supplier, to whom I’d just written a check, had

37

38

Chapter 2

■

Usability and Psychology

a chance of impersonating me, and concluded that asking for the last three
checks’ values would be safer. But the problem I actually had was unexpected.
Having given the checkbook to our accountant for the annual audit, I couldn’t
authenticate myself to get a balance over the phone. There is also a further
liability shift: banks with such systems may expect customers to either keep all
statements securely, or shred them. If someone who steals my physical post
can also steal my money, I’d rather bank elsewhere.
Many e-commerce sites ask for a password explicitly rather than (or as
well as) ‘security questions’ like a maiden name. But the sheer number of
applications demanding a password nowadays exceeds the powers of human
memory. Even though web browsers cache passwords, many customers will
write passwords down (despite being told not to), or use the same password
for many different purposes; relying on your browser cache makes life difficult
when you’re travelling and have to use an Internet café. The upshot is that
the password you use to authenticate the customer of the electronic banking
system you’ve just designed, may be known to a Mafia-operated porn site
as well.
Twenty years ago, when I was working in the banking industry, we security
folks opposed letting customers choose their own PINs for just this sort of
reason. But the marketing folks were in favour, and they won the argument.
Most banks allow the customer to choose their own PIN. It is believed that
about a third of customers use a birthdate, in which case the odds against the
thief are no longer over 3000 to 1 (getting four digits right in three guesses) but
a bit over a hundred to one (and much shorter if he knows the victim). Even if
this risk is thought acceptable, the PIN might still be set to the same value as
the PIN used with a mobile phone that’s shared with family members.
The risk you face as a consumer is not just a direct loss through ‘identity
theft’ or fraud. Badly-designed password mechanisms that lead to password
reuse can cause you to lose a genuine legal claim. For example, if a thief
forges your cash machine card and loots your bank account, the bank will ask
whether you have ever shared your PIN with any other person or company.
If you admit to using the same PIN for your mobile phone, then the bank
can say you were grossly negligent by allowing someone to see you using the
phone, or maybe somebody at the phone company did it — so it’s up to you
to find them and sue them. Eventually, courts may find such contract terms
unreasonable — especially as banks give different and conflicting advice. For
example, the UK bankers’ association has advised customers to change all
their PINs to the same value, then more recently that this is acceptable but
discouraged; their most recent leaflet also suggests using a keyboard pattern
such as ‘C’ (3179) or ‘U’ (1793) [84].
Many attempts to find alternative solutions have hit the rocks. One bank
sent its customers a letter warning them against writing down their PIN, and
instead supplied a distinctive piece of cardboard on which they were supposed

2.4 Passwords

to conceal their PIN in the following way. Suppose your PIN is 2256. Choose
a four-letter word, say ‘blue’. Write these four letters down in the second,
second, fifth and sixth columns of the card respectively, as shown in Figure 2.1.
Then fill up the empty boxes with random letters.
1

2
b
l

3

4

5

6

7

8

9

0

u
e
Figure 2.1: A bad mnemonic system for bank PINs

This is clearly a bad idea. Even if the random letters aren’t written in
a slightly different way, a quick check shows that a four by ten matrix of
random letters may yield about two dozen words (unless there’s an ‘s’ on
the bottom row, when you can get 40–50). So the odds that the thief can
guess the PIN, given three attempts, have just shortened from 1 in 3000-odd
to 1 in 8.

2.4.4.2

Operational Issues

It’s not just the customer end where things go wrong. One important case in
Britain in the late 1980’s was R v Gold and Schifreen. The defendants saw a
phone number for the development system for Prestel (an early public email
service run by British Telecom) in a note stuck on a terminal at an exhibition.
They dialed in later, and found that the welcome screen had an all-powerful
maintenance password displayed on it. They tried this on the live system
too, and it worked! They proceeded to hack into the Duke of Edinburgh’s
electronic mail account, and sent mail ‘from’ him to someone they didn’t like,
announcing the award of a knighthood. This heinous crime so shocked the
establishment that when prosecutors failed to convict the defendants under
the laws then in force, parliament passed Britain’s first specific computer
crime law.
A similar and very general error is failing to reset the default passwords
supplied with certain system services. For example, one top-selling dial access
system in the 1980’s had a default software support user name of 999999 and
a password of 9999. It also had a default supervisor name of 777777 with a
password of 7777. Most sites didn’t change these passwords, and many of
them were hacked once the practice became widely known. Failure to change
default passwords as supplied by the equipment vendor has affected a wide
range of systems. To this day there are web applications running on databases
that use well-known default master passwords — and websites listing the
defaults for everything in sight.

39

40

Chapter 2

2.4.5

■

Usability and Psychology

Social-Engineering Attacks

The biggest practical threat to passwords nowadays is that the user will
break system security by disclosing the password to a third party, whether
accidentally or as a result of deception. This is the core of the ‘phishing’
problem.
Although the first phishing attacks happened in 2003, the word ‘phishing’
itself is older, having appeared in 1996 in the context of the theft of AOL
passwords. Even by 1995, attempts to harvest these to send spam had become
sufficiently common for AOL to have a ‘report password solicitation’ button
on its web page; and the first reference to ‘password fishing’ is in 1990,
in the context of people altering terminal firmware to collect Unix logon
passwords [301]3 .
Phishing brings together several threads of attack technology. The first is
pretexting, which has long been a practical way of getting passwords and
PINs. An old thieves’ trick, having stolen a bank card, is to ring up the victim
and say ‘This is the security department of your bank. We see that your card
has been used fraudulently to buy gold coins. I wonder if you can tell me the
PIN, so I can get into the computer and cancel it?’
There are many variants. A harassed system administrator is called once or
twice on trivial matters by someone who claims to be a very senior manager’s
personal assistant; once he has accepted her as an employee, she calls and
demands a new password for her boss. (See Mitnick’s book [896] for dozens
more examples.) It even works by email. In a systematic experimental study,
336 computer science students at the University of Sydney were sent an email
message asking them to supply their password on the pretext that it was
required to ‘validate’ the password database after a suspected breakin. 138 of
them returned a valid password. Some were suspicious: 30 returned a plausible
looking but invalid password, while over 200 changed their passwords without
official prompting. But very few of them reported the email to authority [556].
Within a tightly-run company, such risks can just about be controlled. We’ve
a policy at our lab that initial passwords are always handed by the sysadmin
to the user on paper. Sun Microsystems had a policy that the root password
for each machine is a 16-character random alphanumeric string, kept in an
envelope with the machine, and which may never be divulged over the phone
or sent over the network. If a rule like this is rigidly enforced throughout an
organization, it will make any pretext attack on a root password conspicuous.
The people who can get at it must be only those who can physically access the
machine anyway. (The problem is of course that you have to teach staff not
3
The first recorded spam is much earlier: in 1865, a London dentist annoyed polite society by
sending out telegrams advertising his practice [415]. Manners and other social mechanisms have
long lagged behind technological progress!

2.4 Passwords

just a rule, but the reasoning behind the rule. Otherwise you end up with the
password stuck to the terminal, as in the Prestel case.)
Another approach, used at the NSA, is to have different colored internal
and external telephones which are not connected to each other, and a policy
that when the external phone in a room is off-hook, classified material can’t
even be discussed in the room — let alone on the phone. A somewhat less
extreme approach (used at our laboratory) is to have different ring tones for
internal and external phones. This works so long as you have alert system
administrators.
Outside of controlled environments, things are harder. A huge problem
is that many banks and other businesses train their customers to act in
unsafe ways. It’s not prudent to click on links in emails, so if you want to
contact your bank you should type in the URL or use a bookmark — yet bank
marketing departments continue to send out emails containing clickable links.
Many email clients — including Apple’s, Microsoft’s, and Google’s — make
plaintext URLs clickable, and indeed their users may never see a URL that
isn’t. This makes it harder for banks to do the right thing.
A prudent customer ought to be cautious if a web service directs him
somewhere else — yet bank systems can use all sorts of strange URLs for their
services. It’s not prudent to give out security information over the phone to
unidentified callers — yet we all get phoned by bank staff who aggressively
demand security information without having any well-thought-out means of
identifying themselves. Yet I’ve had this experience now from two of the banks
with which I’ve done business — once from the fraud department that had got
suspicious about a transaction my wife had made. If even the fraud department
doesn’t understand that banks ought to be able to identify themselves, and
that customers should not be trained to give out security information on the
phone, what hope is there?
You might expect that a dotcom such as eBay would know better, yet its
banking subsidiary PayPal sent its UK customers an email in late 2006 directing
them to a competition at www.paypalchristmas.co.uk, a domain belonging to
a small marketing company I’d never heard of; and despite the fact that they’re
the most heavily phished site on the web, and boast of the technical prowess of
their anti-fraud team when speaking at conferences, the marketing folks seem
to have retained the upper hand over the security folks. In November 2007
they sent an email to a colleague of mine which had a sidebar warning him to
always type in the URL when logging in to his account — and a text body that
asked him to click on a link! (My colleague closed his account in disgust.)
Citibank reassures its customers that it will never send emails to customers to verify personal information, and asks them to disregard and
report emails that ask for such information, including PIN and account
details. So what happened? You guessed it — it sent its Australian customers an email in October 2006 asking customers ‘as part of a security

41

42

Chapter 2

■

Usability and Psychology

upgrade’ to log on to the bank’s website and authenticate themselves
using a card number and an ATM PIN [739]. Meanwhile a marketing spam
from the Bank of America directed UK customers to mynewcard.com. Not
only is spam illegal in Britain, and the domain name inconsistent, and
clickable links a bad idea; but BoA got the certificate wrong (it was for
mynewcard.bankofamerica.com). The ‘mynewcard’ problem had been pointed
out in 2003 and not fixed. Such bad practices are rife among major banks, who
thereby train their customers to practice unsafe computing — by disregarding
domain names, ignoring certificate warnings, and merrily clicking links [399].
As a result, even security experts have difficulty telling bank spam from
phish [301].
But perhaps the worst example of all came from Halifax Share Dealing
Services, part of a large well-known bank in the UK, which sent out a spam
with a URL not registered to the bank. The Halifax’s web page at the time
sensibly advised its customers not to reply to emails, click on links or disclose
details — and the spam itself had a similar warning at the end. The mother of
a student of ours received this spam and contacted the bank’s security department, which told her it was a phish. The student then contacted the ISP to
report abuse, and found that the URL and the service were genuine — although
provided to the Halifax by a third party [842]. When even a bank’s security
department can’t tell spam from phish, how are their customers supposed to?

2.4.6

Trusted Path

The second thread in the background of phishing is trusted path, which refers
to some means of being sure that you’re logging into a genuine machine
through a channel that isn’t open to eavesdropping. Here the deception is
more technical than psychological; rather than inveigling a bank customer into
revealing her PIN to you by claiming to be a policeman, you steal her PIN
directly by putting a false ATM in a shopping mall.
Such attacks go back to the dawn of time-shared computing. A public
terminal would be left running an attack program that looks just like the usual
logon screen — asking for a user name and password. When an unsuspecting
user does this, it will save the password somewhere in the system, reply
‘sorry, wrong password’ and then vanish, invoking the genuine password
program. The user will assume that he made a typing error first time and
think no more of it. This is why Windows has a secure attention sequence,
namely ctrl-alt-del, which is guaranteed to take you to a genuine password
prompt.
If the whole terminal is bogus, then of course all bets are off. We once
caught a student installing modified keyboards in our public terminal room to
capture passwords. When the attacker is prepared to take this much trouble,

2.4 Passwords

then all the ctrl-alt-del sequence achieves is to make his software design
task simpler.
Crooked cash machines and point-of-sale terminals are now a big deal. In
one early case in Connecticut in 1993 the bad guys even bought genuine cash
machines (on credit), installed them in a shopping mall, and proceeded to
collect PINs and card details from unsuspecting bank customers who tried to
use them [33]. Within a year, crooks in London had copied the idea, and scaled
it up to a whole bogus bank branch [635]. Since about 2003, there has been
a spate of skimmers — devices fitted over the front of genuine cash machines
which copy the card data as it’s inserted and use a pinhole camera to record
the customer PIN. Since about 2005, we have also seen skimmers that clip on
to the wires linking point-of-sale terminals in stores to their PIN pads, and
which contain mobile phones to send captured card and PIN data to the crooks
by SMS. (I’ll discuss such devices in much more detail later in the chapter on
Banking and Bookkeeping.)

2.4.7 Phishing Countermeasures
What makes phishing hard to deal with is the combination of psychology and
technology. On the one hand, users have been trained to act insecurely by
their bankers and service providers, and there are many ways in which people
can be conned into clicking on a web link. Indeed much of the marketing
industry is devoted to getting people to click on links. In April 2007 there
was the first reported case of attackers buying Google AdWords in an attempt
to install keyloggers on PCs. This cost them maybe a couple of dollars per
click but enabled them to target the PCs of users thinking of setting up a new
business [1248].
On the other hand, so long as online service providers want to save money
by using the open systems platform provided by web servers and browsers,
the technology does not provide any really effective way for users to identify
the website into which they are about to enter a password.
Anyway, a large number of phishing countermeasures have been tried or
proposed.

2.4.7.1

Password Manglers

A number of people have designed browser plug-ins that take the user-entered
password and transparently turn it into a strong, domain-specific password.
A typical mechanism is to hash it using a secret key and the domain name of
the web site into which it’s being entered [1085]. Even if the user always uses
the same password (even if he uses ‘password’ as his password), each web
site he visits will be provided with a different and hard-to-guess password
that is unique to him. Thus if he mistakenly enters his Citibank password into

43

44

Chapter 2

■

Usability and Psychology

a phishing site, the phisherman gets a different password and cannot use it to
impersonate him.
This works fine in theory but can be tricky to implement in practice. Banks
and shops that use multiple domain names are one headache; another comes
from the different rules that websites use for password syntax (some insist on
alphanumeric, others alpha; some are case sensitive and others not; and so
on). There is also a cost to the user in terms of convenience: roaming becomes
difficult. If only your home machine knows the secret key, then how do you
log on to eBay from a cyber-café when you’re on holiday?

2.4.7.2

Client Certs or Specialist Apps

One of the earliest electronic banking systems I used was one from Bank of
America in the 1980s. This came as a bootable floppy disk; you put it in your PC,
hit ctrl-alt-del, and your PC was suddenly transformed into a bank terminal.
As the disk contained its own copy of the operating system, this terminal was
fairly secure. There are still some online banking systems (particularly at the
corporate end of the market) using such bespoke software. Of course, if a bank
were to give enough customers a special banking application for them to be
a worthwhile target, the phishermen will just tell them to ‘please apply the
attached upgrade’.
A lower-cost equivalent is the client certificate. The SSL protocol supports
certificates for the client as well as the server. I’ll discuss the technical details
later, but for now a certificate is supposed to identify its holder to the other
principals in a transaction and to enable the traffic between them to be securely
encrypted. Server certificates identify web sites to your browser, causing the
lock icon to appear when the name on the certificate corresponds to the name in
the toolbar. Client certificates can be used to make the authentication mutual,
and some UK stockbrokers started using them in about 2006. As of 2007,
the mechanism is still not bulletproof, as certification systems are a pain to
manage, and Javascript can be used to fool common browsers into performing
cryptographic operations they shouldn’t [1163]. Even once that’s fixed, the
risk is that malware could steal them, or that the phisherman will just tell the
customer ‘Your certificates have expired, so please send them back to us for
secure destruction’.

2.4.7.3

Using the Browser’s Password Database

Choosing random passwords and letting your browser cache remember them
can be a pragmatic way of operating. It gets much of the benefit of a password
mangler, as the browser will only enter the password into a web page with the
right URL (IE) or the same hostname and field name (Firefox). It suffers from
some of the same drawbacks (dealing with amazon.com versus amazon.co.uk,

2.4 Passwords

and with roaming). As passwords are stored unencrypted, they are at some
small risk of compromise from malware. Whether you use this strategy may
depend on whether you reckon the greater risk comes from phishing or from
keyloggers. (Firefox lets you encrypt the password database but this is not the
default so many users won’t invoke it.) I personally use this approach with
many low-to-medium-grade web passwords.
Many banks try to disable this feature by setting autocomplete="off" in
their web pages. This stops Firefox and Internet Explorer storing the password.
Banks seem to think this improves security, but I doubt it. There may be a small
benefit in that a virus can’t steal the password from the browser database, but
the phishing defence provided by the browser is disabled — which probably
exposes the customer to much greater risk [913].

2.4.7.4

Soft Keyboards

This was a favorite of banks in Latin America for a while. Rather than using
the keyboard, they would flash up a keyboard on the screen on which the
customer had to type out their password using mouse clicks. The bankers
thought the bad guys would not be able to capture this, as the keyboard could
appear differently to different customers and in different sessions.
However the phishing suppliers managed to write software to defeat it.
At present, they simply capture the screen for 40 pixels around each mouse
click and send these images back to the phisherman for him to inspect and
decipher. As computers get faster, more complex image processing becomes
possible.

2.4.7.5

Customer Education

Banks have put some effort into trying to train their customers to look for
certain features in websites. This has partly been due diligence — seeing to it
that customers who don’t understand or can’t follow instructions can be held
liable — and partly a bona fide attempt at risk reduction. However, the general
pattern is that as soon as customers are trained to follow some particular rule,
the phisherman exploit this, as the reasons for the rule are not adequately
explained.
At the beginning, the advice was ‘Check the English’, so the bad guys either
got someone who could write English, or simply started using the banks’ own
emails but with the URLs changed. Then it was ‘Look for the lock symbol’,
so the phishing sites started to use SSL (or just forging it by putting graphics
of lock symbols on their web pages). Some banks started putting the last four
digits of the customer account number into emails; the phishermen responded
by putting in the first four (which are constant for a given bank and card
product). Next the advice was that it was OK to click on images, but not on

45

46

Chapter 2

■

Usability and Psychology

URLs; the phishermen promptly put in links that appeared to be images but
actually pointed at executables. The advice then was to check where a link
would really go by hovering your mouse over it; the bad guys then either
inserted a non-printing character into the URL to stop Internet Explorer from
displaying the rest, or used an unmanageably long URL (as many banks
also did).
As I remarked earlier, this sort of arms race is most likely to benefit the
attackers. The countermeasures become so complex and counterintuitive that
they confuse more and more users — exactly what the phishermen need. The
safety and usability communities have known for years that ‘blame and train’
is not the way to deal with unusable systems–the only remedy is to make the
systems properly usable in the first place [972].

2.4.7.6

Microsoft Passport

Microsoft Passport was on the face of it a neat idea — a system for using
Microsoft’s logon facilities to authenticate the users of any merchant website.
Anyone with an account on a Microsoft service, such as Hotmail, could log on
automatically to a participating website using a proprietary protocol adapted
from Kerberos to send tickets back and forth in cookies.
One downside was that putting all your eggs in one basket gives people an
incentive to try to kick the basket over. There were many juicy security flaws.
At one time, if you logged in to Passport using your own ID and password,
and then as soon as you’d entered that you backtracked and changed the ID to
somebody else’s, then when the system had checked your password against
the file entry for the first ID, it would authenticate you as the owner of the
second. This is a classic example of a race condition or time-of-check-to-time-ofuse (TOCTTOU) vulnerability, and a spectacular one it was too: anyone in the
world could masquerade as anyone else to any system that relied on Passport
for authentication. Other flaws included cookie-stealing attacks, password
reset attacks and logout failures. On a number of occasions, Microsoft had to
change the logic of Passport rapidly when such flaws came to light. (At least,
being centralised, it could be fixed quickly.)
Another downside came from the business model. Participating sites had
to use Microsoft web servers rather than free products such as Apache,
and it was feared that Microsoft’s access to a mass of data about who
was doing what business with which website would enable it to extend its
dominant position in browser software into a dominant position in the market
for consumer profiling data. Extending a monopoly from one market to
another is against European law. There was an industry outcry that led to the
establishment of the Liberty Alliance, a consortium of Microsoft’s competitors,
which developed open protocols for the same purpose. (These are now used

2.4 Passwords

in some application areas, such as the car industry, but have not caught on for
general consumer use.)

2.4.7.7

Phishing Alert Toolbars

Some companies have produced browser toolbars that use a number of
heuristics to parse URLs and look for wicked ones. Microsoft offers such a
toolbar in Internet Explorer version 7. The idea is that if the user visits a known
phishing site, the browser toolbar turns red; if she visits a suspect site, it turns
yellow; a normal site leaves it white; while a site with an ‘extended validation’
certificate — a new, expensive type of certificate that’s only sold to websites
after their credentials have been checked slightly more diligently than used to
be the case — then it will turn green.
The initial offering has already been broken, according to a paper jointly
authored by researchers from Stanford and from Microsoft itself [650]. Attackers can present users with a ‘picture-in-picture’ website which simply displays
a picture of a browser with a nice green toolbar in the frame of the normal
browser. (No doubt the banks will say ‘maximise the browser before entering your password’ but this won’t work for the reasons discussed above.)
The new scheme can also be attacked using similar URLs: for example,
www.bankofthewest.com can be impersonated as www.bankofthevvest.com.
Even if the interface problem can be fixed, there are problems with using
heuristics to spot dodgy sites. The testing cannot be static; if it were, the phishermen would just tinker with their URLs until they passed the current tests.
Thus the toolbar has to call a server at least some of the time, and check in real
time whether a URL is good or bad. The privacy aspects bear thinking about,
and it’s not entirely clear that the competition-policy issues with Passport have
been solved either.

2.4.7.8

Two-Factor Authentication

Various firms sell security tokens that produce a one-time password. This
can be in response to a challenge sent by the machine to which you want
to log on, or more simply a function of time; you can get a keyfob device
that displays a new eight-digit password every few seconds. I’ll describe the
technology in more detail in the next chapter. These devices were invented in
the early 1980s and are widely used to log on to corporate systems. They are
often referred to as two-factor authentication, as the system typically asks for
a memorised password as well; thus your logon consists of ‘something you
have’ and also ‘something you know’. Password calculators are now used by
some exclusive London private banks, such as the Queen’s bankers, Coutts, to
authenticate their online customers, and we’re now seeing them at a handful
of big money-centre banks too.

47

48

Chapter 2

■

Usability and Psychology

There is some pressure4 for banks to move to two-factor authentication
and issue all their customers with password calculators. But small banks are
chronically short of development resources, and even big banks’ security staff
resist the move on grounds of cost; everyone also knows that the phishermen
will simply switch to real-time man-in-the-middle attacks. I’ll discuss these in
detail in the next chapter, but the basic idea is that the phisherman pretends
to the customer to be the bank and pretends to the bank to be the customer at
the same time, simply taking over the session once it’s established. As of early
2007, only one or two such phishing attacks have been detected, but the attack
technology could be upgraded easily enough.
The favoured two-factor technology in Europe is the chip authentication
program (CAP) device which I’ll also describe in the next chapter. This can
be used either to calculate a logon password, or (once man-in-the-middle
attacks become widespread) to compute a message authentication code on the
actual transaction contents. This means that to pay money to someone you’ll
probably have to type in their account number and the amount twice — once
into the bank’s website and once into your CAP calculator. This will clearly be
a nuisance: tedious, fiddly and error-prone.

2.4.7.9

Trusted Computing

The ‘Trusted Computing’ initiative, which has led to TPM security chips in PC
motherboards, may make it possible to tie down a transaction to a particular
PC motherboard. The TPM chip can support functions equivalent to those
of the CAP device. Having hardware bank transaction support integrated
into the PC will be less fiddly than retyping data at the CAP as well as the PC;
on the other hand, roaming will be a problem, as it is with password manglers
or with relying on the browser cache.
Vista was supposed to ship with a mechanism (remote attestation) that
would have made it easy for bank software developers to identify customer
PCs with high confidence and to stop the bad guys from easily tinkering
with the PC software. However, as I’ll describe later in the chapter on access
control, Microsoft appears to have been unable to make this work yet, so bank
programmers will have to roll their own. As Vista has just been released into
consumer markets in 2007, it may be 2011 before most customers could have
this option available, and it remains to be seen how the banks would cope
with Apple or Linux users. It might be fair to say that this technology has not
so far lived up to the initial hype.
4
In the USA, from the Federal Financial Institutions Examination Council — which, as of
September 2007, 98% of banks were still resisting [1003].

2.4 Passwords

2.4.7.10

Fortified Password Protocols

In 1992, Steve Bellovin and Michael Merritt looked at the problem of how a
guessable password could be used in an authentication protocol between two
machines [158]. They came up with a series of protocols for encrypted key
exchange, whereby a key exchange is combined with a shared password in
such a way that a man-in-the-middle could not guess the password. Various
other researchers came up with other protocols to do the same job.
Some people believe that these protocols could make a significant dent in
the phishing industry in a few years’ time, once the patents run out and the
technology gets incorporated as a standard feature into browsers.

2.4.7.11

Two-Channel Authentication

Perhaps the most hopeful technical innovation is two-channel authentication.
This involves sending an access code to the user via a separate channel, such as
their mobile phone. The Bank of America has recently introduced a version of
this called SafePass in which a user who tried to log on is sent a six-digit code to
their mobile phone; they use this as an additional password [868]. The problem
with this is the same as with the two-factor authentication already tried in
Europe: the bad guys will just use a real-time man-in-the-middle attack.
However, two-channel comes into its own when you authenticate transaction data as well. If your customer tries to do a suspicious transaction, you
can send him the details and ask for confirmation: ‘If you really wanted to
send $7500 to Russia, please enter 4716 now in your browser.’ Implemented
like this, it has the potential to give the level of authentication aimed at by the
CAP designers but with a much more usable interface. Banks have held back
from using two-channel in this way because of worries that usability problems
might drive up their call-centre costs; however the first banks to implement it
report that it hasn’t pushed up support call volumes, and a number of sites
have been implementing it through 2007, with South African banks being in
the forefront. We have already seen the first serious fraud — some Johannesburg crooks social-engineered the phone company to send them a new SIM for
the phone number of the CFO of Ubuntu, a charity that looks after orphaned
and vulnerable children, and emptied its bank account [1017]. The bank and
the phone company are arguing about liability, although the phone company
says it’s fixing its procedures.
Even once the phone-company end of things gets sorted, there are still limits.
Two-channel authentication relies for its security on the independence of the
channels: although the phishermen may be able to infect both PCs and mobile
phones with viruses, so long as both processes are statistically independent,

49

50

Chapter 2

■

Usability and Psychology

only a very small number of people will have both platforms compromised
at the same time. However, if everyone starts using an iPhone, or doing VoIP
telephony over wireless access points, then the assumption of independence
breaks down.
Nonetheless, if I were working for a bank and looking for a front-end
authentication solution today, two-channel would be the first thing I would
look at. I’d be cautious about high-value clients, because of possible attacks
on the phone company, but for normal electronic banking it seems to give the
most bang for the buck.

2.4.8

The Future of Phishing

It’s always dangerous to predict the future, but it’s maybe worth the risk of
wondering where phishing might go over the next seven years. What might I
be writing about in the next edition of this book?
I’d expect to see the phishing trade grow substantially, with attacks on many
non-banks. In November 2007, there was a phishing attack on Salesforce.com
in which the phisherman got a password from a staff member, following which
customers started receiving bogus invoices [614]. If it gets hard to phish the
banks, the next obvious step is to phish their suppliers (such as Salesforce).
In a world of increasing specialisation and outsourcing, how can you track
dependencies and identify vulnerabilities?
Second, research has shown that the bad guys can greatly improve their
yields if they match the context of their phish to the targets [658]; so phish will
get smarter and harder to tell from real emails, just as spam has. Authority
can be impersonated: 80% of West Point cadets bit a phish sent from a
bogus colonel, and a phisherman who uses a social network can do almost
as well: while emails from a random university address got 16% of students
to visit an off-campus website and enter their university password to access
it, this shot up to 72% if the email appeared to come from one of the
target’s friends — with the friendship data collected by spidering open-access
social-networking websites [653]. Future phishermen won’t ask you for your
mother’s maiden name: they’ll forge emails from your mother.
On the technical side, more man-in-the-middle attacks seem likely, as do
more compromises of endpoints such as PCs and mobile phones. If a banking
application running on Vista can only do business on the genuine motherboard,
then the attacker will look for ways to run his software on that motherboard.
If ‘trusted computing’ features in later releases of Vista can stop malware
actually pressing keys and overwriting the screen while a banking application
is running, this might bring real benefits (but I’m not holding my breath).
Starting from the top end of the market, I would not be surprised to see exclusive private banks issuing their customers with dedicated payment devices —
‘Keep your account $50,000 in credit and get a Free Gold Blackberry!’ Such

2.4 Passwords

a device could do wireless payments securely and perhaps even double as a
credit card. (I expect it would fail when the marketing department also decided
it should handle ordinary email, and the crooks figured out ways of pretexting
the rich accountholders into doing things they didn’t really mean to.)
At the middle of the market, I’d expect to see phishing become less distinguishable from more conventional confidence tricks. I mentioned earlier
that the marketing industry nowadays was largely about getting people to
click on links. Now Google has built a twelve-figure business out of this,
so if you’re a crook, why not just advertise there for victims? It’s already
started. And indeed, research by Ben Edelman has found that while 2.73% of
companies ranked top in a web search were bad, 4.44% of companies who
had bought ads from the search engine were bad [416]. (Edelman’s conclusion — ‘Don’t click on ads’ — could be bad news in the medium term for the
search industry.)
On the regulatory side of things, I expect more attempts to interfere in the
identity market, as governments such as America’s and Britain’s look for ways
to issue citizens with identity cards, and as international bodies try to muscle
in. The International Telecommunications Union tried this in 2006 [131]; it
won’t be the last. We will see more pressures to use two-factor authentication,
and to use biometrics. Those parts of the security-industrial complex have
been well fed since 9/11 and will lobby hard for corporate welfare.
However, I don’t believe it will be effective to rely entirely on front-end
controls, whether passwords or fancy widgets. Tricksters will still be able to
con people (especially the old and the less educated), and systems will continue
to get more and more complex, limited only by the security, usability and other
failures inflicted by feature interaction. I believe that the back-end controls will
be at least as important. The very first home banking system — introduced by
the Bank of Scotland in 1984 — allowed payments only to accounts that you
had previously ‘nominated’ in writing. The idea was that you’d write to the
bank to nominate your landlord, your gas company, your phone company
and so on, and then you could pay your bills by email. You set a monthly limit
on how much could be paid to each of them. These early systems suffered
almost no fraud; there was no easy way for a bad man to extract cash. But
the recipient controls were dismantled during the dotcom boom and then
phishing took off.
Some banks are now starting to reintroduce controls — for example, by
imposing a delay and requiring extra authentication the first time a customer
makes a payment to someone they haven’t paid before. Were I designing an
online banking system now, I would invest most of the security budget in
the back end. The phishermen target banks that are slow at recovering stolen
funds [55]. If your asset-recovery team is really on the ball, checks up quickly
on attempts to send money to known cash-extraction channels, claws it back
vigorously, and is ruthless about using the law against miscreants, then the

51

52

Chapter 2

■

Usability and Psychology

phishermen will go after your competitors instead. (I’ll discuss what makes
controls effective later, in the chapter on Banking and Bookkeeping, especially
section 10.3.2.)

2.5 System Issues
Although the fastest-growing public concern surrounding passwords is phishing, and the biggest research topic is psychology, there are a number of other
circumstances in which attackers try to steal or guess passwords, or compromise systems in other ways. There are also technical issues to do with
password entry and storage that I’ll also cover briefly here for the sake of
completeness.
I already noted that the biggest system issue was whether it is possible to
restrict the number of password guesses. Security engineers sometimes refer
to password systems as ‘online’ if guessing is limited (as with ATM PINs) and
‘offline’ if it is not (this originally meant systems where a user could fetch the
password file and take it away to try to guess the passwords of other users,
including more privileged users). The terms are no longer really accurate.
Some offline systems restrict password guesses, such as the smartcards used
in more and more countries for ATMs and retail transactions; these check the
PIN in the smartcard chip and rely on its tamper-resistance to limit guessing.
Many online systems cannot restrict guesses; for example, if you log on using
Kerberos, an opponent who taps the line can observe your key encrypted with
your password flowing from the server to your client, and then data encrypted
with that key flowing on the line; so she can take her time to try out all possible
passwords.
Password guessability is not the only system-level design question, though;
there are others (and they interact). In this section I’ll describe a number of
issues concerning threat models and technical protection, which you might
care to consider next time you design a password system.
Just as we can only talk about the soundness of a security protocol in the
context of a specific threat model, so we can only judge whether a given
password scheme is sound by considering the type of attacks we are trying to
defend against. Broadly speaking, these are:
Targeted attack on one account: an intruder tries to guess a particular
user’s password. He might try to guess the PIN for Bill Gates’s bank
account, or a rival’s logon password at the office, in order to do mischief
directly. When this involves sending emails, it is known as spear phishing.
Attempt to penetrate any account on a system: the intruder tries to get a
logon as any user of the system. This is the classic case of the phisherman
trying to get a password for any user of a target bank’s online service.

2.5 System Issues

Attempt to penetrate any account on any system: the intruder merely
wants an account at any system in a given domain but doesn’t care
which one. Examples are bad guys trying to guess passwords on an
online service so they can send spam from the compromised account,
or use its web space to host a phishing site for a few hours. The modus
operandi is often to try one or two common passwords (such as ‘password1’) on large numbers of randomly-selected accounts. Other possible
attackers might be teens looking for somewhere to hide pornography, or
a private eye tasked to get access to a company’s intranet who is looking
for a beachhead in the form of a logon to some random machine in their
domain.
Service denial attack: the attacker may wish to prevent the legitimate
user from using the system. This might be targeted on a particular account or system-wide.
This taxonomy helps us ask relevant questions when evaluating a password
system.

2.5.1 Can You Deny Service?
Banks often have a rule that a terminal and user account are frozen after three
bad password attempts; after that, an administrator has to reactivate them.
This could be rather dangerous in a military system, as an enemy who got
access to the network could use a flood of false logon attempts to mount a
service denial attack; if he had a list of all the user names on a machine he might
well take it out of service completely. Many commercial websites nowadays
don’t limit guessing because of the possibility of such an attack.
When deciding whether this might be a problem, you have to consider not
just the case in which someone attacks one of your customers, but also the
case in which someone attacks your whole system. Can a flood of false logon
attempts bring down your service? Could it be used to blackmail you? Or
can you turn off account blocking quickly in the event that such an attack
materialises? And if you do turn it off, what sort of attacks might follow?

2.5.2 Protecting Oneself or Others?
Next, to what extent does the system need to protect users from each other?
In some systems — such as mobile phone systems and cash machine systems — no-one should be able to use the service at someone else’s expense.
It is assumed that the attackers are already legitimate users of the system. So
systems are (or at least should be) carefully designed so that knowledge of
one user’s password will not allow another identifiable user’s account to be
compromised.

53

54

Chapter 2

■

Usability and Psychology

Where a user who chooses a password that is easy to guess harms only
himself, a wide variation in password strength can more easily be tolerated.
(Bear in mind that the passwords people choose are very often easy for their
spouses or partners to guess [245]: so some thought needs to be given to issues
such as what happens when a cheated partner seeks vengeance.)
But many systems do not provide strong separation between users. Operating systems such as Unix and Windows may have been designed to protect
one user against accidental interference by another, but they are not hardened to protect against capable malicious actions by other users. They have
many well-publicized vulnerabilities, with more being published constantly
on the web. A competent opponent who can get a single account on a shared
computer system that is not professionally managed can usually become the
system administrator fairly quickly, and from there he can do whatever he
likes.

2.5.3

Attacks on Password Entry

Password entry is often poorly protected.

2.5.3.1

Interface Design

Sometimes the problem is thoughtless interface design. Some common makes
of cash machine had a vertical keyboard at head height, making it simple for
a pickpocket to watch a customer enter her PIN before lifting her purse from
her shopping bag. The keyboards were at a reasonable height for the men who
designed them, but women — and men in many countries — are a few inches
shorter and were highly exposed. One of these machines ‘protected client
privacy’ by forcing the customer to gaze at the screen through a narrow slot.
Your balance was private, but your PIN was not! Many pay-telephones have
a similar problem, and shoulder surfing of calling card details (as it’s known
in the industry) has been endemic at some locations such as major US train
stations and airports.
I usually cover my dialling hand with my body or my other hand when
entering a card number or PIN in a public place — but you shouldn’t design
systems on the assumption that all your customers will do this. Many people
are uncomfortable shielding a PIN from others as it’s a visible signal of distrust;
the discomfort can be particularly acute if someone’s in a supermarket queue
and a friend is standing nearby. In the UK, for example, the banks say that 20%
of users never shield their PIN when entering it, as if to blame any customer
whose PIN is compromised by an overhead CCTV camera [84]; yet in court
cases where I’ve acted as an expert witness, only a few percent of customers
shield their PIN well enough to protect it from an overhead camera. (And just
wait till the bad guys start using infrared imaging.)

2.5 System Issues

2.5.3.2

Eavesdropping

Taking care with password entry may stop the bad guys looking over your
shoulder as you use your calling card at an airport telephone. But it won’t
stop other eavesdropping attacks. The latest modus operandi is for bad people
to offer free WiFi access in public places, and harvest the passwords that
users enter into websites. It is trivial to grab passwords entered into the many
websites that don’t use encryption, and with a bit more work you can get
passwords entered into most of them that do, by using a middleperson attack.
Such attacks have been around for ages. In the old days, a hotel manager
might abuse his switchboard facilities to log the keystrokes you enter at
the phone in your room. That way, he might get a credit card number you
used — and if this wasn’t the card number you used to pay your hotel bill, he
could plunder your account with much less risk. And in the corporate world,
many networked computer systems still send passwords in clear over local
area networks; anyone who can program a machine on the network, or attach
his own sniffer equipment, can harvest them. (I’ll describe in the next chapter
how Windows uses the Kerberos authentication protocol to stop this, and ssh
is also widely used — but there are still many unprotected systems.)

2.5.3.3

Technical Defeats of Password Retry Counters

Many kids find out that a bicycle combination lock can usually be broken
in a few minutes by solving each ring in order of looseness. The same idea
worked against a number of computer systems. The PDP-10 TENEX operating
system checked passwords one character at a time, and stopped as soon as
one of them was wrong. This opened up a timing attack: the attacker would
repeatedly place a guessed password in memory at a suitable location, have it
verified as part of a file access request, and wait to see how long it took to be
rejected [774]. An error in the first character would be reported almost at once,
an error in the second character would take a little longer to report, and in the
third character a little longer still, and so on. So you could guess the characters
once after another, and instead of a password of N characters drawn from an
alphabet of A characters taking AN /2 guesses on average, it took AN/2. (Bear
in mind that in thirty years’ time, all that might remain of the system you’re
building today is the memory of its more newsworthy security failures.)
These same mistakes are being made all over again in the world of embedded
systems. With one remote car locking device, as soon as a wrong byte was
transmitted from the key fob, the red telltale light on the receiver came on. With
some smartcards, it has been possible to determine the customer PIN by trying
each possible input value and looking at the card’s power consumption, then
issuing a reset if the input was wrong. The reason was that a wrong PIN caused
a PIN retry counter to be decremented, and writing to the EEPROM memory

55

56

Chapter 2

■

Usability and Psychology

which held this counter caused a current surge of several milliamps — which
could be detected in time to reset the card before the write was complete [753].
These implementation details matter.

2.5.4

Attacks on Password Storage

Passwords have often been vulnerable where they are stored. There was
a horrendous bug in one operating system update in the 1980s: a user who
entered a wrong password, and was told ‘‘sorry, wrong password’’ merely had
to hit carriage return to get into the system anyway. This was spotted quickly,
and a patch was shipped, but almost a hundred U.S. government systems
in Germany were using unlicensed copies of the software and didn’t get the
patch, with the result that hackers were able to get in and steal information,
which they are rumored to have sold to the KGB.
Another horrible programming error struck a U.K. bank, which issued
all its customers the same PIN by mistake. It happened because the standard
equipment in use at the time for PIN generation required the bank programmer
to first create and store an encrypted PIN, and then use another command to
print out a clear version on a PIN mailer. A bug meant that all customers got
the same encrypted PIN. As the procedures for handling PINs were carefully
controlled, no one in the bank got access to anyone’s PIN other than his or her
own, so the mistake wasn’t spotted until after thousands of customer cards
had been shipped.
Auditing provides another hazard. In systems that log failed password
attempts, the log usually contains a large number of passwords, as users
get the ‘username, password’ sequence out of phase. If the logs are not well
protected then attacks become easy. Someone who sees an audit record of a
failed login with a non-existent user name of e5gv,8yp can be fairly sure that
this string is a password for one of the valid user names on the system.

2.5.4.1

One-Way Encryption

Password storage has also been a problem for some systems. Keeping a
plaintext file of passwords can be dangerous. In MIT’s ‘Compatible Time
Sharing System’, ctss (a predecessor of Multics), it once happened that one
person was editing the message of the day, while another was editing the
password file. Because of a software bug, the two editor temporary files got
swapped, with the result that everyone who logged on was greeted with a
copy of the password file!
As a result of such incidents, passwords are often protected by encrypting
them using a one-way algorithm, an innovation due to Roger Needham and
Mike Guy. The password, when entered, is passed through a one-way function
and the user is logged on only if it matches a previously stored value. However,

2.5 System Issues

it’s often implemented wrong. The right way to do it is to generate a random
salt, hash the password with the salt, and store both the salt and the hash in the
file. The popular blog software Wordpress, as of October 2007, simply stores a
hash of the password — so if the attacker can download the password file for a
Wordpress blog, he can look for weak passwords by comparing the file against
a precomputed file of hashes of words in the dictionary. What’s even worse
is that Wordpress then uses a hash of this hash as the cookie that it sets on
your browser once you’ve logged on. As a result, someone who can look at
the password file can also get in by computing cookies from password hashes,
so he can attack even an adminstrator account with a strong password. In this
case, the one-way algorithm went the wrong way. They should have chosen a
random cookie, and stored a hash of that too.

2.5.4.2

Password Cracking

However, some systems that do use an encrypted password file make it
widely readable (Unix used to be the prime example — the password file was
by default readable by all users). So a user who can fetch this file can then
try to break passwords offline using a dictionary; he encrypts the values in
his dictionary and compares them with those in the file (an activity called
a dictionary attack, or more colloquially, password cracking). The upshot was
that he could impersonate other users, perhaps including a privileged user.
Windows NT was slightly better, but the password file could still be accessed
by users who knew what they were doing.
Most modern operating systems have fixed this problem, but the attack
is still implemented in commercially available password recovery tools. If
you’ve encrypted an Office document with a password you’ve forgotten, there
are programs that will try 350,000 passwords a second [1132]. Such tools can
just as easily be used by a bad man who has got a copy of your data, and
in older systems of your password file. So password cracking is still worth
some attention. Well-designed password protection routines slow down the
guessing by using a complicated function to derive the crypto key from
the password and from a locally-stored salt that changes with each file; the
latest WinZip, for example, allows less than 1000 guesses a second. You can
also complicate a guessing attack by using an odd form of password; most
password guessers try common words first, then passwords consisting of a
root followed by an appendage, such as ‘Kevin06’. Users who avoid such
patterns can slow down the attacker.

2.5.5

Absolute Limits

Regardless of how well passwords are managed, there can be absolute limits
imposed by the design of the platform. For example, Unix systems used to

57

58

Chapter 2

■

Usability and Psychology

limit the length of the password to eight characters (you could often enter more
than this, but the ninth and subsequent characters were ignored). The effort
required to try all possible passwords — the total exhaust time, in cryptanalytic
jargon — is 968 or about 252 , and the average effort for a search is half of this.
A well-financed government agency (or a well-organised hacker group) can
now break any encrypted password in a standard Unix password file.
This motivates more technical defenses against password cracking, including ‘shadow passwords’, that is, encrypted passwords hidden in a private
file (most modern Unices), using an obscure mechanism to do the encryption
(Novell), or using a secret key with the encryption (MVS). The strength of
these mechanisms may vary.
For the above reasons, military system administrators often prefer to issue
random passwords. This also lets the probability of password guessing attacks
be estimated and managed. For example, if L is the maximum password
lifetime, R is login attempt rate, S is the size of the password space, then the
probability that a password can be guessed in its lifetime is P = LR/S, according
to the US Department of Defense password management guideline [377].
There are various problems with this doctrine, of which the worst may be
that the attacker’s goal is often not to guess some particular user’s password
but to get access to any account. If a large defense network has a million
possible passwords and a million users, and the alarm goes off after three
bad password attempts on any account, then the attack is to try one password
for every single account. Thus the quantity of real interest is the probability
that the password space can be exhausted in the lifetime of the system at the
maximum feasible password guess rate.
To take a concrete example, UK government systems tend to issue passwords randomly selected with a fixed template of consonants, vowels and
numbers designed to make them easier to remember, such as CVCNCVCN
(eg fuR5xEb8). If passwords are not case sensitive, the guess probability is
only 214 .52 .102 , or about 229 . So if an attacker could guess 100 passwords a second — perhaps distributed across 10,000 accounts on hundreds of machines
on a network, so as not to raise the alarm — then he’d need about 5 million
seconds, or two months, to get in. With a million-machine botnet, he could
obviously try to speed this up. So if you’re responsible for such a system,
you might find it prudent to do rate control: prevent more than one password guess every few seconds per user account, or (if you can) by source
IP address. You might also keep a count of all the failed logon attempts
and analyse them: is there a constant series of guesses that could indicate an
attempted intrusion? (And what would you do if you noticed one?) With a
commercial website, 100 passwords per second may translate to one compromised user account per second. That may not be a big deal for a web service
with 100 million accounts — but it may still be worth trying to identify the
source of any industrial-scale password-guessing attacks. If they’re from a

2.6 CAPTCHAs

small number of IP addresses, you can block them, but this won’t work so
well if the attacker has a botnet. But if an automated-guessing attack does
emerge, then another way of dealing with it is the CAPTCHA, which I’ll
describe next.

2.6

CAPTCHAs

Recently people have tried to design protection mechanisms that use the
brain’s strengths rather than its weaknesses. One early attempt was Passfaces:
this is an authentication system that presents users with nine faces, only one
of which is of a person they know; they have to pick the right face several
times in a row to log on [356]. The rationale is that people are very good
at recognising other people’s faces, but very bad at describing them: so you
could build a system where it was all but impossible for people to give away
their passwords, whether by accident or on purpose. Other proposals of this
general type have people selecting a series of points on an image — again,
easy to remember but hard to disclose. Both types of system make shoulder
surfing harder, as well as deliberate disclosure offline.
The most successful innovation in this field, however, is the CAPTCHA —
which stands for ‘Completely Automated Public Turing Test to Tell Computers
and Humans Apart.’ You will probably have seen these: they are the little
visual puzzles that you often have to solve to post to a blog, or register for a
free email account. The idea is that a program generates some random text,
and produces a distorted version of it that the user must decipher. Humans
are good at reading distorted text, while programs are less good. CAPTCHAs
first came into use in a big way in 2003 to stop spammers using scripts to open
thousands of accounts on free email services, and their judicious use can make
it a lot harder for attackers to try a few simple passwords with each of a large
number of existing accounts.
The CAPTCHA was devised by Luis von Ahn and colleagues [1304]. It is
inspired by the test famously posed by Alan Turing as to whether a computer
was intelligent, where you put a computer in one room and a human in
another, and invite a human to try to tell them apart. The innovation is that
the test is designed so that a computer can tell the difference between human
and machine, using a known ‘hard problem’ in AI such as the recognition
of distorted text against a noisy background. The idea is that breaking the
CAPTCHA is equivalent to solving the AI problem.
As with all new security technologies, the CAPTCHA is undergoing a period
of rapid coevolution of attack and defence. Many of the image recognition
problems posed by early systems turned out not to be too hard at all. There are
also possible protocol-level attacks; von Ahn mentioned in 2001 that in theory a

59

60

Chapter 2

■

Usability and Psychology

spammer could use a porn site to solve them, by getting people to solve them as
the price of access to free porn [1303]. This has since become a folk legend, and
finally, in October 2007, it actually started to happen: spammers created a game
in which you undress a woman by solving one CAPTCHA after another [134].
Also in that month, we saw the first commercial CAPTCHA-breaking tools
arrive on the market [571].
Finally, the technology can be integrated with authentication and authorisation controls in potentially useful new ways. An interesting example comes
from the banks in Germany, who are introducing an anti-phishing measure
whereby if you authorise a payment online the bank sends you the payee, the
amount and your date of birth, integrated into a CAPTCHA that also contains
a challenge, such as ‘if you want to authorize this payment please enter the
thirteenth password from your list’. This lets them use a static list of one-time
passwords to authenticate actual amounts and beneficiaries, by ensuring that
a real-time man-in-the-middle attack would require a human in the loop. It
may be a better technology than the CAP calculator; it will certainly be less
fiddly than entering transaction details twice. Time will tell if it works.

2.7

Summary

Usability is one of the most important and yet hardest design problems in
many secure systems. It was long neglected as having less techie glamour
then operating systems or cryptographic algorithms; yet most real attacks
nowadays target the user. Phishing is the most rapidly growing threat to
online banking systems, and is starting to be a problem for other sites too.
Other forms of deception are also likely to increase; as technical protection
improves, the bad guys will target the users.
Much of the early work on security usability focused on passwords. Critical
questions to ask when designing a password system include not just whether
people might re-use passwords, but also whether they need to be protected
from each other, whether they can be trained and disciplined, and whether
accounts can be frozen after a fixed number of bad guesses. You also have to
consider whether attackers will target a particular account, or be happy with
breaking any account on a machine or a network; and technical protection
issues such as whether passwords can be snooped by malicious software, false
terminals or network eavesdropping.
However, there is no ‘magic bullet’ in sight. As minor improvements in
protection are devised and fielded, so the phishermen adapt their tactics. At
present, the practical advice is that you should not be a soft touch — harden
your system enough for the phishermen to hit your competitors instead. This
involves not just security usability issues but also your internal controls, which

Further Reading

we will discuss in later chapters. You should assume that some user accounts
will be compromised, and work out how to spot this and limit the damage
when it does happen.

Research Problems
There is a lot of work being done on phishing, but (as we discussed here) none
of it is no far a really convincing solution to the problem. We could do with
some fresh thinking. Are there any neat ways to combine things like passwords,
CAPTCHAs, images and games so as to provide sufficiently dependable twoway authentication between humans and computers? In general, are there any
ways of making middleperson attacks sufficiently harder that it doesn’t matter
if the Mafia owns your ISP?
We also need more fundamental thinking about the relationship between
psychology and security. Between the first edition of this book in 2001 and
the second in 2007, the whole field of security economics sprang into life;
now there are two regular conferences and numerous other relevant events.
So far, security usability is in a fairly embryonic state. Will it also grow big
and prosperous? If so, which parts of existing psychology research will be the
interesting areas to mine?

Further Reading
When I wrote the first edition of this book, there was only a small handful
of notable papers on passwords, including classic papers by Morris and
Thompson [906], Grampp and Morris [550], and Klein [720], and some DoD
guidelines [377]. Since then there has arisen a large research literature on
phishing, with a compendium of papers published as [659]. Perhaps the
greatest gains will come when security engineers start paying attention to
standard HCI texts such as [1039], and researchers start reading widely in the
psychology literature.
A text I’ve found helpful is James Reason’s ‘Human Error’, which essentially
tells us what the safety-critical systems community has learned from many
years studying the cognate problems in their field [1060]. Recently, we’ve
seen the first book on security usability — a collection of the early research
papers [333]. There is also an annual workshop, the Symposium On Usable
Privacy and Security (SOUPS) [1240].
I’m loth to provide much of a guide to the psychology literature, as I don’t
know it as well as I ought to, and we’ve only just started on the project of
building ‘security psychology’ as a discipline. It will take some years for us
to find which psychological theories and experimental results provide us with

61

62

Chapter 2

■

Usability and Psychology

useful insights. But here are some pointers. Tom Gilovich, Dale Griffin and
Danny Kahneman put together a volume of papers summarising the state of
play in the heuristics and biases tradition in 2002 [529]; while a more gentle
introduction might be a book chapter by Richard Samuels, Steven Stich and Luc
Faucher discussing the tensions between that tradition and the evolutionary
psychologists [1106]. It may also be of interest that a number of psychologists
and primatologists (such as Nicholas Humphrey, Richard Byrne and Andy
Whiten) have argued that we evolved intelligence because people who were
better at deception, or at detecting deception in others, had more surviving
offspring — the so-called ‘Machiavellian Brain’ hypothesis [250]. This might
lead us to wonder whether security engineering is the culmination of millions
of years of evolution! (Other psychologists, such as Simon Baron-Cohen,
would deflate any such hubris by arguing that nurturing the young was at
least as important.) Further fascinating analogies with evolutionary biology
have been collected by Raphael Sagarin and Terence Taylor in their book
‘Natural Security’.
Finally, if you’re interested in the dark side, ‘The Manipulation of Human
Behavior’ by Albert Biderman and Herb Zimmer reports experiments on interrogation carried out after the Korean War with US Government funding [162].
It’s also known as the Torturer’s Bible, and describes the relative effectiveness of sensory deprivation, drugs, hypnosis, social pressure and so on when
interrogating and brainwashing prisoners.

CHAPTER

3

Protocols
It is impossible to foresee the consequences of being clever.
— Christopher Strachey
Every thing secret degenerates, even the administration of justice; nothing is safe
that does not show how it can bear discussion and publicity.
— Lord Acton

3.1

Introduction

If security engineering has a deep unifying theme, it is the study of security
protocols. We’ve come across a few protocols informally already — I’ve mentioned challenge-response authentication and Kerberos. In this chapter, I’ll
dig down into the details. Rather than starting off with a formal definition of
a security protocol, I will give a rough indication and then refine it using a
number of examples. As this is an engineering book, I will also give many
examples of how protocols fail.
A typical security system consists of a number of principals such as people,
companies, computers and magnetic card readers, which communicate using
a variety of channels including phones, email, radio, infrared, and by carrying
data on physical devices such as bank cards and transport tickets. The security
protocols are the rules that govern these communications. They are typically
designed so that the system will survive malicious acts such as people telling
lies on the phone, hostile governments jamming radio, or forgers altering
the data on train tickets. Protection against all possible attacks is often too
expensive, so protocols are typically designed under certain assumptions
about the threats. For example, the logon protocol that consists of a user
63

64

Chapter 3

■

Protocols

entering a password into a machine assumes that she can enter it into the right
machine. In the old days of hard-wired terminals in the workplace, this was
reasonable; now that people log on to websites over the Internet, it is much
less so. Evaluating a protocol thus involves answering two questions: first, is
the threat model realistic? Second, does the protocol deal with it?
Protocols may be extremely simple, such as swiping a badge through a
reader in order to enter a building. They often involve interaction, and do
not necessarily involve technical measures like cryptography. For example,
when we order a bottle of fine wine in a restaurant, the standard wine-waiter
protocol provides some privacy (the other diners at our table don’t learn the
price), some integrity (we can be sure we got the right bottle and that it wasn’t
switched for, or refilled with, cheap plonk) and non-repudiation (it’s hard for
the diner to complain afterwards that the wine was off). Blaze gives other
examples from applications as diverse as ticket inspection, aviation security
and voting in [185].
At the technical end of things, protocols can be much more complex. The
world’s bank card payment system has dozens of protocols specifying how
customers interact with cash machines and retail terminals, how a cash machine
or terminal talks to the bank that operates it, how the bank communicates with
the network operator, how money gets settled between banks, how encryption
keys are set up between the various cards and machines, and what sort of
alarm messages may be transmitted (such as instructions to capture a card).
All these protocols have to work together in a large and complex system.
Often a seemingly innocuous design feature opens up a serious flaw. For
example, a number of banks encrypted the customer’s PIN using a key known
only to their central computers and cash machines, and wrote it to the card
magnetic strip. The idea was to let the cash machine verify PINs locally, which
saved on communications and even allowed a limited service to be provided
when the cash machine was offline. After this system had been used for many
years without incident, a programmer (who was playing around with a card
reader used in a building access control system) discovered that he could
alter the magnetic strip of his own bank card by substituting his wife’s bank
account number for his own. He could then take money out of her account
using the modified card and his own PIN. He realised that this enabled him
to loot any other customer’s account too, and went on to steal hundreds of
thousands over a period of years. The affected banks had to spend millions
on changing their systems. And some security upgrades can take years; at
the time of writing, much of Europe has moved from magnetic-strip cards to
smartcards, while America has not. Old and new systems have to work side
by side so that European cardholders can buy from American stores and vice
versa. This also opens up opportunities for the crooks; clones of European
cards are often used in magnetic-strip cash machines in other countries, as the
two systems’ protection mechanisms don’t quite mesh.

3.2 Password Eavesdropping Risks

So we need to look systematically at security protocols and how they fail. As
they are widely deployed and often very badly designed, I will give a number
of examples from different applications.

3.2

Password Eavesdropping Risks

Passwords and PINs are still the foundation on which much of computer
security rests, as they are the main mechanism used to authenticate humans
to machines. I discussed their usability and ‘human interface’ problems of
passwords in the last chapter. Now let us consider some more technical
attacks, of the kind that we have to consider when designing more general
protocols that operate between one machine and another. A good case study
comes from simple embedded systems, such as the remote control used to open
your garage or to unlock the doors of cars manufactured up to the mid-1990’s.
These primitive remote controls just broadcast their serial number, which also
acts as the password.
An attack that became common was to use a ‘grabber’, a device that would
record a code broadcast locally and replay it later. These devices, seemingly
from Taiwan, arrived on the market in about 1995; they enabled thieves lurking
in parking lots to record the signal used to lock a car door and then replay it
to unlock the car once the owner had left1 .
One countermeasure was to use separate codes for lock and unlock. But
this is still not ideal. First, the thief can lurk outside your house and record
the unlock code before you drive away in the morning; he can then come
back at night and help himself. Second, sixteen-bit passwords are too short.
It occasionally happened that people found they could unlock the wrong car
by mistake (or even set the alarm on a car whose owner didn’t know he
had one [217]). And by the mid-1990’s, devices appeared which could try all
possible codes one after the other. A code will be found on average after about
215 tries, which at ten per second takes under an hour. A thief operating in a
parking lot with a hundred vehicles within range would be rewarded in less
than a minute with a car helpfully flashing its lights.
So another countermeasure was to double the length of the password from
16 to 32 bits. The manufacturers proudly advertised ‘over 4 billion codes’. But
this only showed they hadn’t really understood the problem. There was still
1

With garage doors it’s even worse. A common chip is the Princeton PT2262, which uses 12
tri-state pins to encode 312 or 531,441 address codes. However implementers often don’t read
the data sheet carefully enough to understand tri-state inputs and treat them as binary instead,
getting 212 . Many of them only use eight inputs, as the other four are on the other side of the
chip. And as the chip has no retry-lockout logic, an attacker can cycle through the combinations
quickly and open your garage door after 27 attempts on average.

65

66

Chapter 3

■

Protocols

only one code (or two codes) for each car, and although guessing was now
impractical, grabbers still worked fine.
Using a serial number as a password has a further vulnerability: there may
be many people with access to it. In the case of a car, this might mean all the
dealer staff, and perhaps the state motor vehicle registration agency. Some
burglar alarms have also used serial numbers as master passwords, and here
it’s even worse: the serial number may appear on the order, the delivery note,
the invoice and all the other standard commercial paperwork.
Simple passwords are sometimes the appropriate technology, even when
they double as serial numbers. For example, my monthly season ticket for
the swimming pool simply has a barcode. I’m sure I could make a passable
forgery with our photocopier and laminating machine, but as the turnstile is
attended and the attendants get to know the ‘regulars’, there is no need for
anything more expensive. My card keys for getting into the laboratory where
I work are slightly harder to forge: the one for student areas uses an infrared
barcode, while the card for staff areas has an RFID chip that states its serial
number when interrogated over short-range radio. Again, these are probably
quite adequate — our more expensive equipment is in rooms with fairly good
mechanical door locks. But for things that lots of people want to steal, like cars,
a better technology is needed. This brings us to cryptographic authentication
protocols.

3.3

Who Goes There? — Simple Authentication

A simple example of an authentication device is an infrared token used in some
multistorey parking garages to enable subscribers to raise the barrier. This first
transmits its serial number and then sends an authentication block consisting
of the same serial number, followed by a random number, all encrypted using
a key which is unique to the device. We will postpone discussion of how to
encrypt data and what properties the cipher should have; we will simply use
the notation {X}K for the message X encrypted under the key K.
Then the protocol between the access token in the car and the parking garage
can be written as:
T−
→ G : T, {T, N}KT
This is the standard protocol engineering notation, and can be a bit confusing
at first, so we’ll take it slowly.
The in-car token sends its name T followed by the encrypted value of
T concatenated with N, where N stands for ‘number used once’, or nonce.
Everything within the braces is encrypted, and the encryption binds T and
N together as well as obscuring their values. The purpose of the nonce is
to assure the recipient that the message is fresh, that is, it is not a replay of

3.3 Who Goes There? — Simple Authentication

an old message that an attacker observed. Verification is simple: the parking
garage server reads T, gets the corresponding key KT, deciphers the rest of the
message, checks that the nonce N has not been seen before, and finally that
the plaintext contains T (which stops a thief in a car park from attacking all
the cars in parallel with successive guessed ciphertexts).
One reason many people get confused is that to the left of the colon, T
identifies one of the principals (the token which represents the subscriber)
whereas to the right it means the name (that is, the serial number) of the token.
Another is that once we start discussing attacks on protocols, we can suddenly
start finding that the token T’s message intended for the parking garage G was
actually intercepted by the freeloader F and played back at some later time. So
the notation is unfortunate, but it’s too well entrenched now to change easily.
Professionals often think of the T −
→ G to the left of the colon is simply a hint
as to what the protocol designer had in mind.
The term nonce can mean anything that guarantees the freshness of a
message. A nonce can, according to the context, be a random number, a serial
number, a random challenge received from a third party, or even a timestamp.
There are subtle differences between these approaches, such as in the level
of resistance they offer to various kinds of replay attack, and they increase
system complexity in different ways. But in very low-cost systems, the first
two predominate as it tends to be cheaper to have a communication channel
in one direction only, and cheap devices usually don’t have clocks.
Key management in such devices can be very simple. In a typical garage
token product, each token’s key is simply its serial number encrypted under a
global master key KM known to the central server:
KT = {T}KM
This is known as key diversification. It’s a common way of implementing
access tokens, and is very widely used in smartcard-based systems as well.
But there is still plenty of room for error. One old failure mode that seems
to have returned is for the serial numbers not to be long enough, so that
someone occasionally finds that their remote control works for another car in
the car park as well. Having 128-bit keys doesn’t help if the key is derived by
encrypting a 16-bit serial number.
Weak ciphers also turn up. One token technology used by a number of car
makers in their door locks and immobilisers employs a block cipher known as
Keeloq, which was designed in the late 1980s to use the minimum number of
gates; it consists of a large number of iterations of a simple round function.
However in recent years an attack has been found on ciphers of this type, and
it works against Keeloq; it takes about an hour’s access to your key to collect
enough data for the attack, and then about a day on a PC to process it and
recover the embedded cryptographic key [172]. You might not think this a
practical attack, as someone who gets access to your key can just drive off with

67

68

Chapter 3

■

Protocols

your car. However, in some implementations, there is also a terrible protocol
vulnerability, in that the key diversification is not done using the block cipher
itself, but using exclusive-or: KT = T ⊕ KM. So once you have broken a single
vehicle key for that type of car, you can immediately work out the key for any
other car of that type. The researchers who found this attack suggested ‘Soon,
cryptographers will drive expensive cars.’
Indeed protocol vulnerabilities usually give rise to more, and simpler,
attacks than cryptographic weaknesses do. At least two manufacturers have
made the mistake of only checking that the nonce is different from last time,
so that given two valid codes A and B, the series ABABAB... was interpreted
as a series of independently valid codes. A thief could open a car by replaying
the last-but-one code. A further example comes from the world of prepayment
utility meters. Over a million households in the UK, plus many millions in
developing countries, have an electricity or gas meter that accepts encrypted
tokens; the householder buys a token, takes it home and inserts it into the
meter, which then dispenses the purchased quantity of energy. One electricity
meter widely used in South Africa checked only that the nonce in the decrypted
command was different from last time. So the customer could charge the meter
up to the limit by buying two low-value power tickets and then repeatedly
feeding them in one after the other [59].
So the question of whether to use a random number or a counter is not as easy
as it might seem [316]. If you use random numbers, the lock has to remember
a reasonable number of past codes. You might want to remember enough of
them to defeat the valet attack. Here, someone who has temporary access to the
token — such as a valet parking attendant — can record a number of access
codes and replay them later to steal your car. Providing enough nonvolatile
memory to remember hundreds or even thousands of old codes might push
you to a more expensive microcontroller, and add a few cents to the cost of
your lock.
If you opt for counters, the problem is synchronization. The key may be
used for more than one lock; it may also be activated repeatedly by jostling
against something in your pocket (I once took an experimental token home
where it was gnawed by my dogs). So there has to be a way to recover after the
counter has been incremented hundreds or possibly even thousands of times.
This can be turned to advantage by allowing the lock to ‘learn’, or synchronise
on, a key under certain conditions; but the details are not always designed
thoughtfully. One common product uses a sixteen bit counter, and allows
access when the deciphered counter value is the last valid code incremented
by no more than sixteen. To cope with cases where the token has been used
more than sixteen times elsewhere (or gnawed by a family pet), the lock will
open on a second press provided that the counter value has been incremented

3.3 Who Goes There? — Simple Authentication

between 17 and 32,767 times since a valid code was entered (the counter rolls
over so that 0 is the successor of 65,535). This is fine in many applications, but
a thief who can get six well-chosen access codes — say for values 0, 1, 20,000,
20,001, 40,000 and 40,001 — can break the system completely. So you would
have to think hard about whether your threat model includes a valet able to
get access codes corresponding to chosen counter values, either by patience or
by hardware hacking.
A recent example of design failure comes from TinyOS, an operating system
used in sensor networks based on the IEEE 802.15.4 ad-hoc networking
standard. The TinySec library commonly used for security protocols contains
not one, but three counters. The first is lost as the radio chip driver overwrites
it, the second isn’t remembered by the receiver, and although the third is
functional, it’s used for reliability rather than security. So if someone monkeys
with the traffic, the outcome is ‘error’ rather than ‘alarm’, and the network will
resynchronise itself on a bad counter [340].
So designing even a simple token authentication mechanism is not at all
straightforward. There are many attacks that do not involve ‘breaking’ the
encryption. Such attacks are likely to become more common as cryptographic
authentication mechanisms proliferate, many of them designed by programmers who thought the problem was easy and never bothered to read a book
like this one. And there are capable agencies trying to find ways to defeat
these remote key entry systems; in Thailand, for example, Muslim insurgents
use them to detonate bombs, and the army has responded by deploying
jammers [1000].
Another important example of authentication, and one that’s politically contentious for different reasons, is ‘accessory control’. Many printer companies
embed authentication mechanisms in printers to ensure that genuine toner
cartridges are used. If a competitor’s product is loaded instead, the printer
may quietly downgrade from 1200 dpi to 300 dpi, or simply refuse to work at
all. Mobile phone vendors make a lot of money from replacement batteries,
and now use authentication protocols to spot competitors’ products so they
can be blocked or even drained more quickly. All sorts of other industries are
getting in on the act; there’s talk in the motor trade of cars that authenticate
their major spare parts. I’ll discuss this in more detail in Chapter 22 along
with copyright and rights management generally. Suffice it to say here that
security mechanisms are used more and more to support business models,
by accessory control, rights management, product tying and bundling. It is
wrong to assume blindly that security protocols exist to keep ‘bad’ guys ‘out’.
They are increasingly used to constrain the lawful owner of the equipment in
which they are built; their purpose may be of questionable legality or contrary
to public policy.

69

70

Chapter 3

3.3.1

■

Protocols

Challenge and Response

Most cars nowadays have remote-controlled door unlocking, though most
also have a fallback metal key to ensure that you can still get into your car
even if the RF environment is noisy. Many also use a more sophisticated twopass protocol, called challenge-response, to actually authorise engine start. As
the car key is inserted into the steering lock, the engine controller sends a
challenge consisting of a random n-bit number to the key using short-range
radio. The car key computes a response by encrypting the challenge. So,
writing E for the engine controller, T for the transponder in the car key, K
for the cryptographic key shared between the transponder and the engine
controller, and N for the random challenge, the protocol may look something
like:
E−
→T:
T−
→E:

N
{T, N}K

This is still not bulletproof.
In one system, the random numbers generated by the engine management
unit turned out to be predictable, so it was possible for a thief to interrogate the
key in the car owner’s pocket, as he passed, with the anticipated next challenge.
In fact, many products that incorporate encryption have been broken at some
time or another because their random number generators weren’t random
enough [533, 395]. The fix varies from one application to another. It’s possible
to build hardware random number generators using radioactive decay, but
this isn’t common because of health and safety concerns. There are various
sources of usable randomness in large systems such as PCs, such as the small
variations in the rotation speed of the hard disk caused by air turbulence [358].
PC software products often mix together the randomness from a number of
environmental sources such as network traffic and keystroke timing and from
internal system sources [567]; and the way these sources are combined is often
critical [703]. But in a typical embedded system such as a car lock, the random
challenge is generated by encrypting a counter using a special key which is
kept inside the device and not used for any other purpose.
Locks are not the only application of challenge-response protocols. In HTTP
Digest Authentication, a web server challenges a client or proxy, with whom
it shares a password, by sending it a nonce. The response consists of the
hash of the nonce, the password, and the requested URI [493]. This provides a
mechanism that’s not vulnerable to password snooping. It’s used, for example,
to authenticate clients and servers in SIP, the protocol for Voice-Over-IP
(VOIP) telephony. It is much better than sending a password in the clear,
but suffers from various weaknesses — the most serious being middleperson
attacks, which I’ll discuss shortly.

3.3 Who Goes There? — Simple Authentication

A much more visible use of challenge-response is in two-factor authentication.
Many organizations issue their staff with password generators to let them
log on to corporate computer systems [1354]. These may look like calculators
(and some even function as calculators) but their main function is as follows.
When you want to log in to a machine on the network, you call up a logon
screen and are presented with a random challenge of maybe seven digits. You
key this into your password generator, together with a PIN of maybe four
digits. The device encrypts these eleven digits using a secret key shared with
the corporate security server, and displays the first seven digits of the result.
You enter these seven digits as your password. This protocol is illustrated in
Figure 3.1. If you had a password generator with the right secret key, and you
entered the PIN right, and you typed in the result correctly, then the corporate
computer system lets you in. But if you do not have a genuine password
generator for which you know the PIN, your chance of logging on is small.
Formally, with S for the server, P for the password generator, PIN for the
user’s Personal Identification Number that bootstraps the password generator,
U for the user and N for the random nonce:
S−
→U:
U−
→P:
P−
→U:
U−
→S:

N
N, PIN
{N, PIN}K
{N, PIN}K

N?

N, PIN

N?

.....

Figure 3.1: Password generator use

K

71

72

Chapter 3

■

Protocols

These devices appeared from the early 1980s and caught on first with phone
companies, then in the 1990s with banks for use by staff. There are simplified
versions that don’t have a keyboard, but just generate a new access code every
minute or so by encrypting a counter: the RSA SecurID is the best known.
One sector after another has been adopting authentication tokens of one kind
or another to replace or supplement passwords; the US Defense Department
announced in 2007 that the introduction of an authentication system based
on the DoD Common Access Card had cut network intrusions by 46% in the
previous year [225].
The technology is now starting to spread to the customer side of things. By
2001, password generators were used by some exclusive private banks, such
as Coutts, to authenticate their online customers. These banks never suffered
any phishing fraud. By 2006, some banks in the Netherlands and Scandinavia
had rolled out the technology to all their millions of customers; then the frauds
started. The phishermen typically use real-time man-in-the-middle attacks
(which I’ll describe in the next section) to take over a session once the user
has authenticated herself to the bank. As of late 2007, some banks in the
UK and elsewhere in Europe have been introducing the Chip Authentication
Program (CAP), which is implemented by giving bank customers a calculator
that uses their bank card to do crypto2 . This calculator, when loaded with a
bank card, will ask for the customer’s PIN and, if it’s entered correctly, will
compute a response code based on either a counter (as a one-off authentication
code for a card transaction, or a one-step logon to a banking website) or a
challenge (for a two-step logon). There is also a third mode of operation: if
session takeover becomes a problem, the CAP calculator can also be used to
authenticate transaction data. In this case, it’s planned to have the customer
enter the amount and the last eight digits of the payee account number into
her CAP calculator.
But the result might not be as good in banking as it has been in the armed
forces. First, when your wallet is stolen the thief might be able to read your
PIN digits from the calculator — they will be the dirty and worn keys. If you
just use one bank card, then the thief’s chance of guessing your PIN in 3 tries
has just come down from about 1 in 3000 to about 1 in 10. Second, when you
use your card in a Mafia-owned shop (or in a shop whose terminals have been
quietly reprogrammed without the owner’s knowledge), the bad guys have
everything they need to loot your account. Not only that — they can compute
a series of CAP codes to give them access in the future, and use your account
for wicked purposes such as money laundering. Third, someone who takes
your bank card from you at knifepoint can now verify that you’ve told them
2
Bank cards in many European countries have an EMV smartcard chip on them, and new UK
bank cards have software to compute authentication codes as well as to operate ATMs and shop
terminals.

3.3 Who Goes There? — Simple Authentication

the right PIN. A further problem is that the mechanisms can be used in a
range of protocols; if you have to give a one-off authentication code over the
phone to buy a book with your bank card, and the bookseller can then use
that code to log on to your bank, it’s clearly a bad thing. A deeper problem
is that once lots of banks use one-time passwords, the phishermen will just
rewrite their scripts to do real-time man-in-the-middle attacks. These have
already been used against the early adopter banks in the Netherlands and
Scandinavia. To see how they work, we will now look at a military example.

3.3.2

The MIG-in-the-Middle Attack

The ever-increasing speeds of warplanes in the 1930s and 1940s, together
with the invention of the jet engine, radar and rocketry, made it ever more
difficult for air defence forces to tell their own craft apart from the enemy’s. This
led to a serious risk of ‘fratricide’ — people shooting down their colleagues
by mistake — and drove the development of systems to ‘identify-friend-orfoe’ (IFF). These were first fielded in World War II, and in their early form
enabled an airplane illuminated by radar to broadcast an identifying number
to signal friendly intent. In 1952, this system was adopted to identify civil
aircraft to air traffic controllers and, worried about the loss of security once
it became widely used, the U.S. Air Force started a research programme to
incorporate cryptographic protection in the system. Nowadays, the typical air
defense system sends random challenges with its radar signals, and friendly
aircraft have equipment and keys that enable them to identify themselves
with correct responses. The chapter on electronic warfare has more details on
modern systems.
It’s tricky to design a good IFF system. One of the problems is illustrated
by the following story, which I heard from an officer in the South African
Air Force (SAAF). After it was published in the first edition of this book,
the story was disputed — as I’ll discuss below. Be that as it may, similar
games have been played with other electronic warfare systems since World
War 2. The ‘Mig-in-the-middle’ story has in any event become part of the
folklore, and it nicely illustrates how attacks can be carried out in real time on
challenge-response authentication protocols.
In the late 1980’s, South African troops were fighting a war in northern
Namibia and southern Angola. The goals were to keep Namibia under white
rule, and impose a client government (UNITA) on Angola. Because the South
African Defence Force consisted largely of conscripts from a small white
population, it was important to limit casualties, so most South African soldiers
remained in Namibia on policing duties while the fighting to the north was
done by UNITA troops. The role of the SAAF was twofold: to provide tactical
support to UNITA by bombing targets in Angola, and to ensure that the
Angolans and their Cuban allies did not return the compliment in Namibia.

73

74

Chapter 3

■

Protocols

Suddenly, the Cubans broke through the South African air defenses and
carried out a bombing raid on a South African camp in northern Namibia,
killing a number of white conscripts. This proof that their air supremacy had
been lost helped the Pretoria government decide to hand over Namibia to the
insurgents — itself a huge step on the road to majority rule in South Africa
several years later. The raid may also have been the last successful military
operation ever carried out by Soviet bloc forces.
Some years afterwards, a SAAF officer told me how the Cubans had pulled
it off. Several MIGs had loitered in southern Angola, just north of the South
African air defense belt, until a flight of SAAF Impala bombers raided a target
in Angola. Then the MIGs turned sharply and flew openly through the SAAF’s
air defenses, which sent IFF challenges. The MIGs relayed them to the Angolan
air defense batteries, which transmitted them at a SAAF bomber; the responses
were relayed back in real time to the MIGs, who retransmitted them and were
allowed through — as in Figure 3.2. According to my informant, this had a
significant effect on the general staff in Pretoria. Being not only outfought by
black opponents, but actually outsmarted, was not consistent with the world
view they had held up till then.
After this tale was published in the first edition of my book, I was contacted
by a former officer in SA Communications Security Agency who disputed the
story’s details. He said that their IFF equipment did not use cryptography
yet at the time of the Angolan war, and was always switched off over enemy
territory. Thus, he said, any electronic trickery must have been of a more
primitive kind. However, others tell me that ‘Mig-in-the-middle’ tricks were
significant in Korea, Vietnam and various Middle Eastern conflicts.
In any case, the tale illustrates the basic idea behind an attack known
to the cryptographic community as the man-in-the-middle or (more recently)
the middleperson attack. It applies in a straightforward way to the challengeresponse authentication performed by password calculators: the phishing site
invites the mark to log on and simultaneously opens a logon session with his
bank. The bank sends a challenge; the phisherman relays this to the mark,
who uses his device to respond to it; the phisherman relays it to the bank,
and is now authenticated to the bank as the mark. This is why, as I discussed
above, European banks are introducing not just a simple response to a single
challenge, but an authentication code based on input fields such as the amount,
the payee account number and a transaction sequence number.
However, once the protocol-level vulnerabilities are fixed by including all
the transaction data, the big problem will be usability. If it takes two minutes
and the entry of dozens of digits to make a payment, then a lot of customers
will get digits wrong, give up, and then either call the call center or send paper
checks — undermining the cost savings of online banking. Also, the bad guys
will be able to exploit the fallback mechanisms, perhaps by spoofing customers

3.3 Who Goes There? — Simple Authentication

SAAF
N?
N

ANGOLA

N

K

K

N?

MIG
N?

N

K

SAAF

NAMIBIA

Figure 3.2: The MIG-in-the middle attack

into calling voice phishing phone numbers that run a middleperson attack
between the customer and the call center.
We will come across the man-in-the-middle attack again and again in
applications ranging from pay-TV to Internet security protocols. It even
applies in online gaming. As the mathematician John Conway once remarked,
it’s easy to get at least a draw against a grandmaster at postal chess: just play
two grandmasters at once, one as white and the other as black, and relay the
moves between them!
In many cases, middleperson attacks are possible but not economic. In the
case of car keys, it should certainly be possible to steal a car by having an
accomplice follow the driver and electronically relay the radio challenge to
you as you work the lock. (One of our students has actually demonstrated

75

76

Chapter 3

■

Protocols

this for our RFID door locks.) But, for the average car thief, it would be a lot
simpler to just pick the target’s pocket or mug him.
In early 2007, it became clear that there is a practical middleperson attack on
the protocols used by the EMV smartcards issued to bank customers in Europe.
A bad man could build a wicked terminal that masqueraded, for example, as
a parking meter; when you entered your card and PIN to pay a £2.50 parking
fee, the transaction could be relayed to a crook loitering near a self-service
terminal in a hardware store, who would use a card emulator to order goods.
When you get your statement, you might find you’ve been debited £2,500 for
a wide-screen TV [915]. The basic problem here is the lack of a trustworthy
user interface on the card; the cardholder doesn’t really know which terminal
his card is doing business with. I’ll discuss such attacks further in the chapter
on Banking and Bookkeeping.

3.3.3

Reflection Attacks

Further interesting problems arise with mutual authentication, that is, when
two principals have to identify each other. Suppose, for example, that a simple challenge-response IFF system designed to prevent anti-aircraft gunners
attacking friendly aircraft had to be deployed in a fighter-bomber too. Now
suppose that the air force simply installed one of their air gunners’ challenge
units in each aircraft and connected it to the fire-control radar. But now an
enemy bomber might reflect a challenge back at our fighter, get a correct
response, and then reflect that back as its own response:
F−
→B:N
B−
→F:N
F−
→ B : {N}K
B−
→ F : {N}K
So we will want to integrate the challenge system with the response generator. It is still not enough just for the two units to be connected and share a list
of outstanding challenges, as an enemy attacked by two of our aircraft might
reflect a challenge from one of them to be answered by the other. It might also
not be acceptable to switch manually from ‘attack’ to ‘defense’ during combat.
There are a number of ways of stopping this ‘reflection attack’: in many cases,
it is sufficient to include the names of the two parties in the authentication
exchange. In the above example, we might require a friendly bomber to reply
to the challenge:
F−
→B:N

3.3 Who Goes There? — Simple Authentication

with a response such as:
B−
→ F : {B, N}K
Thus a reflected response {F, N} (or even {F , N} from the fighter pilot’s
wingman) could be detected.
This is a much simplified account of IFF, but it serves to illustrate the
subtelty of the trust assumptions that underlie an authentication protocol. If
you send out a challenge N and receive, within 20 milliseconds, a response
{N}K , then — since light can travel a bit under 3,730 miles in 20 ms — you
know that there is someone with the key K within 2000 miles. But that’s all you
know. If you can be sure that the response was not computed using your own
equipment, you now know that there is someone else with the key K within two
thousand miles. If you make the further assumption that all copies of the key
K are securely held in equipment which may be trusted to operate properly,
and you see {B, N}K , you might be justified in deducing that the aircraft with
callsign B is within 2000 miles. A clear understanding of trust assumptions
and their consequences is at the heart of security protocol design.
By now you might think that the protocol design aspects of IFF have been
exhaustively discussed. But we’ve omitted one of the most important problems — and one which the designers of early IFF systems did not anticipate. As
radar returns are weak, the signal from the IFF transmitter on board an aircraft
will often be audible at a much greater range than the return. The Allies learned
this the hard way; in January 1944, decrypts of Enigma messages revealed that
the Germans were plotting British and American bombers at twice the normal
radar range by interrogating their IFF. So many modern systems authenticate
the challenge as well as the response. The NATO mode XII, for example, has
a 32 bit encrypted challenge, and a different valid challenge is generated for
every interrogation signal, of which there are typically 250 per second. Theoretically there is no need to switch off over enemy territory, but in practice an
enemy who can record valid challenges can replay them as part of an attack.
Relays are also possible, as with the Mig in the middle.
Many other IFF design problems are less protocol-related, such as the
difficulties posed by neutrals, error rates in dense operational environments,
how to deal with equipment failure, how to manage keys, and how to cope
with multinational coalitions such as that put together for Operation Desert
Storm. I’ll return to IFF in Chapter 19. For now, the spurious-challenge problem
serves to reinforce an important point: that the correctness of a security protocol
depends on the assumptions made about the requirements. A protocol that
can protect against one kind of attack (being shot down by your own side) but
which increases the exposure to an even more likely attack (being shot down
by the other side) does more harm than good. In fact, the spurious-challenge
problem became so serious in World War II that some experts advocated
abandoning IFF altogether, rather than taking the risk that one bomber pilot

77

78

Chapter 3

■

Protocols

in a formation of hundreds would ignore orders and leave his IFF switched on
while over enemy territory.

3.4

Manipulating the Message

We’ve now seen a number of middleperson attacks that reflect or spoof the
information used to authenticate a participant’s identity — from ATM cards
that could be reprogrammed to ‘identify’ the wrong customer, to attacks on
IFF. However, there are more complex attacks where the attacker does not just
obtain false identification, but manipulates the message content in some way.
An example is when dishonest cabbies insert pulse generators in the cable
that connects their taximeter to a sensor in their taxi’s gearbox. The sensor
sends pulses as the prop shaft turns, which lets the meter work out how far
the taxi has gone. A pirate device, which inserts extra pulses, makes the taxi
appear to have gone further. We’ll discuss such attacks at much greater length
in the chapter on ‘Monitoring Systems’, in section 12.3.
Another example is a key log attack which defeated many pay-TV systems
in Europe in the 1990s and still appears to work in China. The attack is also
known as delayed data transfer, or DDT. First-generation pay-TV equipment
has a decoder, which deciphers the video signal, and a customer smartcard
which generates the deciphering keys. These keys are recomputed every few
hundred milliseconds by using a one-way encryption function applied to
various ‘entitlement control messages’ that appear in the signal. Such systems
can be very elaborate (and we’ll discuss some more complex attacks on them
later) but there is a very simple attack which works against a lot of them. If the
messages that pass between the smartcard and the decoder are the same for
all decoders (which is usually the case) then a subscriber can log all the keys
sent by his card to his decoder and post it online somewhere. People without a
subscription, but who have video-recorded the enciphered program, can then
download the key log and use it to decipher the tape.
Changing pay-TV protocols to prevent DDT attacks can be difficult. The
base of installed equipment is huge, and many of the obvious countermeasures
have an adverse effect on legitimate customers (such as by preventing them
videotaping movies). Pay-TV companies generally ignore this attack, since
connecting a PC up to a satellite TV decoder through a special hardware
adaptor is something only hobbyists do; it is too inconvenient to be a real
threat to their revenue stream. In the rare cases where it becomes a nuisance,
the strategy is usually to identify the troublesome subscribers and send
entitlement control messages that deactivate their cards.
Message-manipulation attacks aren’t limited to ‘consumer’ grade systems.
The Intelsat satellites used for international telephone and data traffic have
robust mechanisms to prevent a command being accepted twice — otherwise

3.5 Changing the Environment

an attacker could repeatedly order the same maneuver to be carried out until
the satellite ran out of fuel [1027].

3.5

Changing the Environment

A very common cause of protocol failure is that the environment changes, so
that assumptions which were originally true no longer hold and the security
protocols cannot cope with the new threats.
One nice example comes from the ticketing systems used by the urban
transport authority in London. In the early 1980’s, passengers devised a
number of scams to cut the cost of commuting. For example, a passenger
who commuted a long distance from a suburban station to downtown might
buy two cheaper, short distance season tickets — one between his suburban
station and a nearby one, and the other between his destination and another
downtown station. These would let him get through the barriers, and on the
rare occasions he was challenged by an inspector in between, he would claim
that he’d boarded at a rural station which had a broken ticket machine.
A large investment later, the system had all the features necessary to stop
such scams: all barriers were automatic, tickets could retain state, and the laws
had been changed so that people caught without tickets got fined on the spot.
But suddenly the whole environment changed, as the national transport
system was privatized to create dozens of rail and bus companies. Some of the
new operating companies started cheating each other, and there was nothing
the system could do about it! For example, when a one-day travel pass was
sold, the revenue was distributed between the various bus, train and subway
operators using a formula that depended on where it was sold. Suddenly,
the train companies had a motive to book all their ticket sales through the
outlet that let them keep the largest percentage. As well as bad outsiders
(passengers), we now had bad insiders (rail companies), and the design just
hadn’t allowed for them. Chaos and litigation ensued.
The transport system’s problem was not new; it had been observed in the
Italian ski resort of Val di Fassa in the mid-1970’s. There, one could buy a
monthly pass for all the ski lifts in the valley. An attendant at one of the lifts
was observed with a deck of cards, one of which he swiped through the reader
between each of the guests. It turned out that the revenue was divided up
between the various lift operators according to the number of people who had
passed their turnstiles. So each operator sought to inflate its own figures as
much as it could [1217].
Another nice example comes from the world of cash machine fraud. In 1993
and 1994, Holland suffered an epidemic of ‘phantom withdrawals’; there was
much controversy in the press, with the banks claiming that their systems
were secure while many people wrote in to the papers claiming to have been

79

80

Chapter 3

■

Protocols

cheated. Eventually the banks were shamed into actively investigating the
claims, and noticed that many of the victims had used their bank cards at a
certain filling station near Utrecht. This was staked out and one of the staff
was arrested. It turned out that he had tapped the line from the card reader
to the PC that controlled it; his tap recorded the magnetic stripe details from
their cards while he used his eyeballs to capture their PINs [33].
Why had the system been designed so badly? Well, when the standards
for managing magnetic stripe cards and PINs were developed in the early
1980’s by organizations such as IBM and VISA, the engineers had made two
assumptions. The first was that the contents of the magnetic strip — the card
number, version number and expiration date — were not secret, while the
PIN was [880]. (The analogy used was that the magnetic strip was your name
and the PIN your password. I will have more to say on the difficulties of
naming below.) The second assumption was that bank card equipment would
only be operated in trustworthy environments, such as in a physically robust
automatic teller machine, or by a bank clerk at a teller station. So it was ‘clearly’
only necessary to encrypt the PIN, on its way from the PIN pad to the server;
the magnetic strip data could be sent in clear from the card reader.
Both of these assumptions had changed by 1993. An epidemic of card
forgery, mostly in the Far East in the late 1980’s, drove banks to introduce
authentication codes on the magnetic strips. Also, the commercial success of
the bank card industry led banks in many countries to extend the use of debit
cards from ATMs to terminals in all manner of shops. The combination of these
two environmental changes undermined the original system design: instead of
putting a card whose magnetic strip contained no security data into a trusted
machine, people were putting a card with security data in clear on the strip
into an untrusted machine. These changes had come about so gradually, and
over such a long period, that the industry didn’t see the problem coming.

3.6

Chosen Protocol Attacks

Some firms are trying to sell the idea of a ‘multifunction smartcard’ — an
authentication device that could be used in a wide range of transactions
to save you having to carry around dozens of different cards and keys.
Governments keen to push ID cards in the wake of 9/11 have tried to get them
used for many other transactions; some want a single card to be used for ID,
banking and even transport ticketing. Singapore went so far as to experiment
with a bank card that doubled as military ID. This introduced some interesting
new risks: if a Navy captain tries to withdraw some cash from an ATM after a
good dinner and forgets his PIN, will he be unable to take his ship to sea until
Monday morning when they open the bank and give him his card back?

3.6 Chosen Protocol Attacks

Picture 143!

Buy 10 gold coins

Prove your age

Sign ‘X’

by signing ‘X’
Customer

sigK X

Mafia porn
site

sigK X

BANK

Figure 3.3: The Mafia-in-the-middle attack

Suppose that the banks in Europe were to introduce the CAP protocol to get
their customers to authenticate themselves to electronic banking websites, but
rather than forcing their customers to fiddle about with a calculator device they
just issued all customers with smartcard readers that could be attached to their
PC. This would certainly improve convenience and usability. You might think
it would improve security too; the EMV protocol enables the card to calculate
a message authentication code (MAC) on transaction data such as the amount,
merchant number, date and transaction serial number. Message manipulation
attacks against electronic banking payments would be prevented.
Or would they? The idea behind the ‘Chosen Protocol Attack’ is that given a
target protocol, you design a new protocol that will attack it if the users can be
inveigled into reusing the same token or crypto key. So how might the Mafia
design a protocol to attack CAP?
Here’s one approach. It used to be common for people visiting a porn
website to be asked for ‘proof of age,’ which usually involves giving a
credit card number, whether to the site itself or to an age checking service.
If credit and debit cards become usable in PCs, it would be natural for
the porn site to ask the customer to authenticate a random challenge as
proof of age. A porn site can then mount a ‘Mafia-in-the-middle’ attack as
shown in Figure 3.3. They wait until an unsuspecting customer visits their site,
then order something resellable (such as gold coins) from a dealer, playing
the role of the coin dealer’s customer. When the coin dealer sends them the
transaction data for authentication, they relay it through their porn site to
the waiting customer. The poor man OKs it, the Mafia gets the gold coins, and
when thousands of people suddenly complain about the huge charges to their
cards at the end of the month, the porn site has vanished — along with the
gold [702].
This is a more extreme variant on the Utrecht scam, and in the 1990s
a vulnerability of this kind found its way into international standards: the
standards for digital signature and authentication could be run back-to-back
in this way. It has since been shown that many protocols, though secure in
themselves, can be broken if their users can be inveigled into reusing the same
keys in other applications [702]. This is why, for CAP to be secure, it may

81

82

Chapter 3

■

Protocols

well have to be implemented in a stand-alone device into which the customer
enters all the transaction parameters directly. Even so, some way has to be
found to make it hard for the phishermen to trick the customer into computing
an authentication code on data that they supply to the victim. The use of the
customer’s bank card in the CAP calculator may at least help to bring home
that a banking transaction is being done.
In general, using crypto keys (or other authentication mechanisms) in
more than one application is dangerous, while letting other people bootstrap
their own application security off yours can be downright foolish. If a bank
lets its smartcards also be used to load credit into prepayment electricity
meters, it would have to worry very hard about whether bad software could
be used in electricity vending stations (or even electricity meters) to steal
money. Even if those risks could be controlled somehow, liability issues can
arise from unplanned or emergent dependencies. A bank that changed its card
specification might break the metering system — leaving its customers literally
in the dark and risking a lawsuit from the power company. If the bank heeds
these risks and tests system changes properly with all the dependant systems,
then changes will be much more expensive. Crooks who hack the bank could
black out the neighbourhood. The bank might still want to take this risk,
though, reckoning that power company customers would be locked in more
tightly to the bank, enabling it to charge them more. Security dependencies
can have all sorts of strange effects, and we will return to this subject again
and again later.

3.7

Managing Encryption Keys

The examples of security protocols that we have discussed so far are mostly
about authenticating a principal’s name, or application data such as the
impulses driving a taximeter. There is one further class of authentication
protocols that is very important — the protocols used to manage cryptographic
keys. Until recently, such protocols were largely used in the background to
support other operations; much of the technology was developed to manage
the keys used by cash machines and banks to communicate with each other.
But now, systems such as pay-TV use key management to control access to the
system directly.
Authentication protocols are now also used in distributed computer systems
for general key management purposes, and are therefore becoming ever more
important. Kerberos was the first such system to come into widespread use,
and a variant of it is used in Windows. I’ll now lay the foundations for an
understanding of Kerberos.

3.7 Managing Encryption Keys

3.7.1 Basic Key Management
The basic idea behind key distribution protocols is that where two principals want to communicate, they may use a trusted third party to effect an
introduction.
When discussing authentication protocols, it is conventional to give the
principals human names in order to avoid getting lost in too much algebraic
notation. So we will call the two communicating principals ‘Alice’ and ‘Bob’,
and the trusted third party ‘Sam’. But please don’t assume that we are
talking about human principals. Alice and Bob are likely to be programs
while Sam is a server; for example, Alice might be a program in a taximeter,
Bob the program in a gearbox sensor and Sam the computer at the taxi
inspection station.
Anyway, a simple authentication protocol could run as follows.
1. Alice first calls Sam and asks for a key for communicating with Bob.
2. Sam responds by sending Alice a pair of certificates. Each contains a copy
of a key, the first encrypted so only Alice can read it, and the second
encrypted so only Bob can read it.
3. Alice then calls Bob and presents the second certificate as her introduction.
Each of them decrypts the appropriate certificate under the key they share
with Sam and thereby gets access to the new key. Alice can now use the
key to send encrypted messages to Bob, and to receive messages from him
in return.
Replay attacks are a known problem with authentication protocols, so in
order that both Bob and Alice can check that the certificates are fresh, Sam may
include a timestamp in each of them. If certificates never expire, there might
be serious problems dealing with users whose privileges have been revoked.
Using our protocol notation, we could describe this as
A→S:
S→A:
A→B:

A, B
{A, B, KAB , T}KAS , {A, B, KAB , T}KBS
{A, B, KAB , T}KBS , {M}KAB

Expanding the notation, Alice calls Sam and says she’d like to talk to
Bob. Sam makes up a session key message consisting of Alice’s name, Bob’s
name, a key for them to use, and a timestamp. He encrypts all this under
the key he shares with Alice, and he encrypts another copy of it under the
key he shares with Bob. He gives both ciphertexts to Alice. Alice retrieves
the key from the ciphertext that was encrypted to her, and passes on to
Bob the ciphertext encrypted for him. She now sends him whatever message
she wanted to send, encrypted using this key.

83

84

Chapter 3

3.7.2

■

Protocols

The Needham-Schroeder Protocol

Many things can go wrong, and here is a famous historical example. Many
existing key distribution protocols are derived from the Needham-Schroeder
protocol, which appeared in 1978 [960]. It is somewhat similar to the above,
but uses nonces rather than timestamps. It runs as follows:
Message 1
Message 2
Message 3
Message 4
Message 5

A→S:
S→A:
A→B:
B→A:
A→B:

A, B, NA
{NA , B, KAB , {KAB , A}KBS }KAS
{KAB , A}KBS
{NB }KAB
{NB − 1}KAB

Here Alice takes the initiative, and tells Sam: ‘I’m Alice, I want to talk to Bob,
and my random nonce is NA .’ Sam provides her with a session key, encrypted
using the key she shares with him. This ciphertext also contains her nonce so
she can confirm it’s not a replay. He also gives her a certificate to convey this
key to Bob. She passes it to Bob, who then does a challenge-response to check
that she is present and alert.
There is a subtle problem with this protocol — Bob has to assume that the
key KAB he receives from Sam (via Alice) is fresh. This is not necessarily so:
Alice could have waited a year between steps 2 and 3. In many applications
this may not be important; it might even help Alice to cache keys against
possible server failures. But if an opponent — say Charlie — ever got hold of
Alice’s key, he could use it to set up session keys with many other principals.
Suppose, for example, that Alice had also asked for and received a key to
communicate with Dave, and after Charlie stole her key he sent messages to
Sam pretending to be Alice and got keys for Freddie and Ginger. He might
also have observed message 2 in her protocol exchanges with Dave. So now
Charlie could impersonate Alice to Dave, Freddie and Ginger. So when Alice
finds out that her key has been stolen, perhaps by comparing message logs
with Dave, she’d have to get Sam to contact everyone for whom she’d ever
been issued a key, and tell them that her old key was no longer valid. She could
not do this herself as she doesn’t know anything about Freddie and Ginger. In
other words, revocation is a problem: Sam may have to keep complete logs of
everything he’s ever done, and these logs would grow in size forever unless
the principals’ names expired at some fixed time in the future.
Almost 30 years later, this example still generates controversy in the security
protocols community. The simplistic view is that Needham and Schroeder just
got it wrong; the view argued by Susan Pancho and Dieter Gollmann (for
which I have much sympathy) is that this is one more example of a protocol
failure brought on by shifting assumptions [538, 1002]. 1978 was a kinder,
gentler world; computer security then concerned itself with keeping ‘bad
guys’ out, while nowadays we expect the ‘enemy’ to be the users of the

3.7 Managing Encryption Keys

system. The Needham-Schroeder paper explicitly assumes that all principals
behave themselves, and that all attacks come from outsiders [960]. With these
assumptions, the protocol remains sound.

3.7.3

Kerberos

An important practical derivative of the Needham-Schroeder protocol may be
found in Kerberos, a distributed access control system that originated at MIT
and is now one of the standard authentication tools in Windows [1224]. Instead
of a single trusted third party, Kerberos has two kinds: an authentication server
to which users log on, and a ticket granting server which gives them tickets
allowing access to various resources such as files. This enables more scalable
access management. In a university, for example, one might manage students
through their halls of residence but manage file servers by departments; in
a company, the personnel people might register users to the payroll system
while departmental administrators manage resources such as servers and
printers.
First, Alice logs on to the authentication server using a password. The client
software in her PC fetches a ticket from this server that is encrypted under her
password and that contains a session key KAS . Assuming she gets the password
right, she now controls KAS and to get access to a resource B controlled by the
ticket granting server S, the following protocol takes place. Its outcome is a
key KAB with timestamp TS and lifetime L, which will be used to authenticate
Alice’s subsequent traffic with that resource:
A→S:
S→A:
A→B:
B→A:

A, B
{TS , L, KAB , B, {TS , L, KAB , A}KBS }KAS
{TS , L, KAB , A}KBS , {A, TA }KAB
{TA + 1}KAB

Translating this into English: Alice asks the ticket granting server for access
to B. If this is permissible, the ticket {TS , L, KAB , A}KBS is created containing a
suitable key KAB and given to Alice to use. She also gets a copy of the key
in a form readable by her, namely encrypted under KAS . She now verifies the
ticket by sending a timestamp TA to the resource, which confirms it’s alive by
sending back the timestamp incremented by one (this shows it was able to
decrypt the ticket correctly and extract the key KAB ).
The vulnerability of Needham-Schroeder has been fixed by introducing
timestamps rather than random nonces. But, as in most of life, we get little
in security for free. There is now a new vulnerability, namely that the clocks
on our various clients and servers might get out of synch; they might even be
desynchronized deliberately as part of a more complex attack.

85

86

Chapter 3

3.7.4

■

Protocols

Practical Key Management

So we can use a protocol like Kerberos to set up and manage working keys
between users given that each user shares one or more long-term keys with
a server that acts as a key distribution centre. I’ll describe a number of
other similar protocols later; for example, in the chapter on ‘Banking and
Bookkeeping’ I’ll discuss how a bank can set up a long-term key with each of
its ATMs and with each of the interbank networks with which it’s associated.
The bank then uses protocols not too unlike Kerberos to establish a ‘key of
the day’ with each ATM and with each network switch; so when you turn up
at the ATM belonging to a foreign bank and ask for money from your own
bank via the Cirrus network, the ATM will encrypt the transaction using the
working key it shares with the bank that owns it, and the bank will then pass
on the transaction to Cirrus encrypted with the key of the day for that network.
So far so good. But a moment’s thought will reveal that the bank has to
maintain several keys for each of the several hundred ATMs that it owns — a
long-term master key, plus perhaps an encryption key and an authentication
key; several keys for each of the several dozen bank networks of which it’s a
member; passwords and other security information for each of several million
electronic banking customers, and perhaps keys for them as well if they’re
given client software that uses cryptography. Oh, and there may be encrypted
passwords for each of several thousand employees, which might also take the
form of Kerberos keys encrypted under user passwords. That’s a lot of key
material. How is it to be managed?
Key management is a complex and difficult business and is often got
wrong because it’s left as an afterthought. A good engineer will sit down
and think about how many keys are needed, how they’re to be generated,
how long they need to remain in service and how they’ll eventually be
destroyed. There is a much longer list of concerns — many of them articulated
in the Federal Information Processing Standard for key management [948]. In
addition, things go wrong as applications evolve; it’s important to provide extra
keys to support next year’s functionality, so that you don’t compromise your
existing ones by reusing them in protocols that turn out to be incompatible.
It’s also important to support recovery from security failure. Yet there are no
standard ways of doing either.
As for practical strategies, there are a number — none of them straightforward. Public-key crypto, which I’ll discuss in Chapter 5, can slightly simplify
the key management task. Long-term keys can be split into a private part and a
public part; you don’t have to keep the public part secret (as its name implies)
but you do have to guarantee its integrity. In banking the usual answer is
to use dedicated cryptographic processors called security modules, which I’ll
describe in detail in the chapter on ‘Tamper Resistance’. These do all the cryptography and contain internal keys with which application keys are protected.

3.8 Getting Formal

Thus you get your security module to generate master keys for each of your
ATMs; you store their encrypted values in your ATM master file. Whenever
a transaction comes in from that ATM, you retrieve the encrypted key from
the file and pass it to the security module along with the encrypted data. The
module then does what’s necessary: it decrypts the PIN and verifies it, perhaps
against an encrypted value kept locally. Unfortunately, the protocols used to
set all this up are also liable to failure. Many attacks have been found that
exploit the application programming interface, or API, of the security module,
where these protocols are exposed. I will describe these attacks in detail in
the chapter on API Security. For now, it’s enough to note that getting security
protocols right is hard. You should not design them at home, any more than
you design your own explosives.

3.8

Getting Formal

Subtle difficulties of the kind we have seen with the above protocols, and the
many ways in which protection properties depend on quite subtle starting
assumptions that protocol designers may get wrong (or that may be misunderstood later), have led researchers to apply formal methods to key distribution
protocols. The goal of this exercise was originally to decide whether a protocol
was right or wrong: it should either be proved correct, or an attack should be
exhibited. More recently this has expanded to clarifying the assumptions that
underlie a given protocol.
There are a number of different approaches to verifying the correctness
of protocols. The best known is the logic of belief, or BAN logic, named after
its inventors Burrows, Abadi and Needham [249]. It reasons about what a
principal might reasonably believe having seen of certain messages, timestamps and so on. A second is the random oracle model, which I touch on in the
chapter on cryptology and which is favored by people working on the theory
of cryptography; this appears less expressive than logics of belief, but can tie
protocol properties to the properties of the underlying encryption algorithms.
Finally, a number of researchers have applied mainstream formal methods
such as CSP and verification tools such as Isabelle.
Some history exists of flaws being found in protocols that had been
proved correct using formal methods; the following subsection offers a
typical example.

3.8.1 A Typical Smartcard Banking Protocol
The COPAC system is an electronic purse used by VISA in countries with poor
telecommunications [48]. It was the first live financial system whose underlying protocol suite was designed and verified using such formal techniques, and

87

88

Chapter 3

■

Protocols

in particular a variant of the BAN logic. A similar protocol is now used in the
‘Geldkarte,’ an electronic purse issued by banks in Germany, and adopted also
by French banks as ‘Moneo’. There’s also a system in Belgium called ‘Proton’.
The European applications focus on low-value transactions with devices such
as parking meters and vending machines for which it may not be economical
to provide a network connection.
Transactions take place from a customer smartcard to a merchant smartcard
(which in the case of a vending machine is kept in the machine and changed
when it’s replenished). The customer gives the merchant an electronic check
with two authentication codes on it; one that can be checked by the network,
and one that can be checked by the customer’s bank. A simplified version of
the protocol is as follows.
C−
→R:
R−
→C:
C−
→R:

{C, NC }K
{R, NR , C, NC }K
{C, NC , R, NR , X}K

In English: the customer and the retailer share a key K. Using this key, the
customer encrypts a message containing its account number C and a customer
transaction serial number NC . The retailer confirms its own account number
R and his own transaction serial number NR , as well as the information it’s
just received from the customer. The customer now sends the electronic check
X, along with all the data exchanged so far in the protocol. One can think of
the electronic check as being stapled to a payment advice with the customer’s
and retailer’s account numbers and their respective reference numbers. (The
reason for repeating all previous data in each message is to prevent message
manipulation attacks using cut-and-paste.)

3.8.2

The BAN Logic

The BAN logic provides a formal method for reasoning about the beliefs
of principals in cryptographic protocols. Its underlying idea is that we will
believe that a message is authentic if it is encrypted with a relevant key and it
is also fresh (that is, generated during the current run of the protocol). Further
assumptions include that principals will only assert statements they believe
in, and that some principals are authorities for certain kinds of statement. This
is formalized using a notation which includes:
A |≡ X A believes X, or, more accurately, that A is entitled to believe X;
A |∼ X A once said X (without implying that this utterance was recent or not);
A |⇒ X A has jurisdiction over X, in other words A is the authority on X and is
to be trusted on it;

3.8 Getting Formal

A  X A sees X, that is, someone sent a message to A containing X in such a
way that he can read and repeat it;
X X is fresh, that is, contains a current timestamp or some information
showing that it was uttered by the relevant principal during the current
run of the protocol;
{X}K X encrypted under the key K, as in the rest of this chapter;
A ↔K B A and B share the key K, in other words it is an appropriate key for
them to use to communicate.
There are further symbols dealing, for example, with public key operations
and with passwords, that need not concern us here.
These symbols are manipulated using a set of postulates which include:
the message meaning rule states that if A sees a message encrypted under K,
and K is a good key for communicating with B, then he will believe that the
message was once said by B. (We assume that each principal can recognize
A |≡ A ↔K B, A  {X}K
and ignore his or her own messages.) Formally,
A |≡ B |∼ X
the nonce-verification rule states that if a principal once said a message,
and the message is fresh, then that principal still believes it. Formally,
A |≡ X, A |≡ B |∼ X
A |≡ B |≡ X
the jurisdiction rule states that if a principal believes something, and is an
authority on the matter, then he or she should be believed. Formally, we
A |≡ B |⇒ X, A |≡ B |≡ X
write that
A |≡ X
In this notation, the statements on the top are the conditions, and the one on
the bottom is the result. There are a number of further rules to cover the more
mechanical aspects of manipulation; for example, if A sees a statement then
he sees its components provided he knows the necessary keys, and if part of a
formula is known to be fresh, then the whole formula must be.

3.8.3 Verifying the Payment Protocol
Assuming that the key K is only available to principals who can be trusted to
execute the protocol faithfully, formal verification is now straightforward. The
trick is to start from the desired result and work backwards. In this case, we
wish to prove that the retailer should trust the check, i.e., R |≡ X (the syntax
of checks and cryptographic keys is similar for our purposes here; a check is
good if and only if it is genuine and the date on it is sufficiently recent).

89

90

Chapter 3

■

Protocols

Now R |≡ X will follow under the jurisdiction rule from R |≡ C |⇒ X (R
believes C has jurisdiction over X) and R |≡ C |≡ X (R believes C believes X).
The former condition follows from the hardware constraint, that no-one
except C could have uttered a text of the form {C, . . .}K .
The latter, that R |≡ C |≡ X, must be deduced using the nonce verification
rule from X (X is fresh) and R |≡ C |∼ X (R believes C uttered X).
X follows from its occurrence in {C, NC , R, NR , X}K which contains the
sequence number NR , while R |≡ C |∼ X follows from the hardware constraint.
The above summary of the proof is, of necessity, telegraphic. If you want to
understand logics of authentication in detail, you should consult the original
papers [48] and see the recommendations for further reading at the end of this
chapter.

3.8.4

Limitations of Formal Verification

Formal methods can be an excellent way of finding bugs in security protocol
designs as they force the designer to make everything explicit and thus
confront difficult design choices that might otherwise be fudged. However,
they have their limitations, too.
One problem is in the external assumptions we make. For example, we
assumed that the key wasn’t available to anyone who might use it in an
unauthorized manner. In practice, this is not always true. Although our purse
protocol is executed in tamper-resistant smartcards, their software can have
bugs, and in any case the tamper-resistance they offer is never complete. (I’ll
discuss this in the chapter on Tamper Resistance.) So the system has various
fallback mechanisms to detect and react to card forgery, such as shadow
accounts which track the amount of money that should be on each card and
which are updated as transactions are cleared. It also has lists of hot cards that
are distributed to terminals; these are needed anyway for stolen cards, and
can be used for forged cards too.
Second, there are often problems with the idealisation of the protocol. An
interesting flaw was found in an early version of this system. The key K actually
consisted of two keys — the encryption was done first with a ‘transaction key’
which was diversified (that is, each card had its own variant) and then again
with a ‘bank key’, which was not diversified. The former was done by the
network operator, and the latter by the bank which issued the card. The
reasons for this included dual control, and to ensure that even if an attacker
managed to drill the keys out of a single card, he would only be able to forge
that card, not make forgeries which would pass as other cards (and thus defeat
the hot card mechanism). But since the bank key was not diversified, it must
be assumed to be known to any attacker who has broken a card. This means
that he can undo the outer wrapping of encryption, and in some circumstances
message replay was possible. (The bank key was diversified in a later version
before any villains discovered and exploited the flaw.)

3.9 Summary

In this case there was no failure of the formal method, as no attempt was ever
made to verify the diversification mechanism. But it does illustrate a common
problem in security engineering — that vulnerabilities arise at the boundary
between two protection technologies. In this case, there were three technologies: the hardware tamper resistance, the authentication protocol and the
shadow account / hot card list mechanisms. Different protection technologies
are often the domain of different experts who don’t completely understand the
assumptions made by the others. (In fact, that’s one reason security engineers
need a book such as this one: to help subject specialists understand each others’
tools and communicate with each other more effectively.)
For these reasons, people have explored alternative ways of assuring the
design of authentication protocols, including the idea of protocol robustness. Just
as structured programming techniques aim to ensure that software is designed
methodically and nothing of importance is left out, so robust protocol design is
largely about explicitness. Robustness principles include that the interpretation
of a protocol should depend only on its content, not its context; so everything
of importance (such as principals’ names) should be stated explicitly in the
messages. There are other issues concerning the freshness provided by serial
numbers, timestamps and random challenges, and on the way encryption
is used. If the protocol uses public key cryptography or digital signature
mechanisms, there are further more technical robustness issues.

3.9

Summary

Passwords are just one (simple) example of a more general concept, the
security protocol. Protocols specify the series of steps that principals use
to establish trust relationships in a system, such as authenticating a claim
to identity, demonstrating ownership of a credential, or granting a claim
on a resource. Cryptographic authentication protocols, whether one-pass
(e.g., using random nonces) or two-pass (challenge-response) are used for
a wide range of such purposes, from basic entity authentication to provide
infrastructure for distributed systems that allows trust to be taken from where it
exists to where it is needed. Security protocols are fielded in all sorts of systems
from remote car door locks through military IFF systems to authentication in
distributed computer systems.
It is difficult to design effective security protocols. They suffer from a
number of potential problems, including middleperson attacks, modification
attacks, reflection attacks, and replay attacks. These threats can interact with
implementation vulnerabilities such as poor random number generators. Using
mathematical techniques to verify the correctness of protocols can help, but it
won’t catch all the bugs. Some of the most pernicious failures are caused by
creeping changes in the environment for which a protocol was designed, so
that the protection it gives is no longer adequate.

91

92

Chapter 3

■

Protocols

Research Problems
At several times during the past 20 years, some people have thought that
protocols had been ‘done’ and that we should turn to new research topics.
They have been repeatedly proved wrong by the emergence of new protocol
applications with a new crop of errors and attacks to be explored. Formal
methods blossomed in the early 1990s, then key management protocols; during
the mid-1990’s the flood of proposals for electronic commerce mechanisms
kept us busy; and in the later 1990’s a whole series of mechanisms proposed
for protecting copyright on the Internet provided us with targets. Since
2000, one strand of protocol research has acquired an economic flavour as
security mechanisms are used more and more to support business models; the
designer’s ‘enemy’ is often a commercial competitor, or even the customer.
Another has applied protocol analysis tools to look at the security of application
programming interfaces (APIs), a topic to which I’ll return later.
Will people continue to develop faulty protocols which other people attack,
or will we manage to develop a methodology for designing them right first
time? What are the exact uses and limitations of formal methods, and other
mathematical approaches such as the random oracle model?
At the system level, how do we manage the tension between the principle
that robust protocols are generally those in which everything is completely
specified and checked (principals’ names, roles, security policy statement,
protocol version, time, date, sequence number, security context, maker of
grandmother’s kitchen sink) and the system engineering principle that a good
specification should not overconstrain the implementer?

Further Reading
Research papers on security protocols are scattered fairly widely throughout
the literature. The main introductory papers to read are probably the original
Needham-Schroeder paper [960]; the Burrows-Abadi-Needham authentication logic [249]; papers by Abadi and Needham, and Anderson and Needham,
on protocol robustness [2, 73]; and there is a survey paper by Anderson and
Needham [74]. In [707] there is an analysis of a defective security protocol, carried out using three different formal methods. Beyond that, the proceedings of
the security protocols workshops [290, 291] provide leads to current research,
and there are many papers scattered around a wide range of conferences.

CHAPTER

4

Access Control
Going all the way back to early time-sharing systems we systems
people regarded the users, and any code they wrote, as the mortal
enemies of us and each other. We were like the police force
in a violent slum.
— Roger Needham
Microsoft could have incorporated effective security measures as
standard, but good sense prevailed. Security systems have a nasty
habit of backfiring and there is no doubt they would cause
enormous problems.
— Rick Maybury

4.1

Introduction

Access control is the traditional center of gravity of computer security. It is
where security engineering meets computer science. Its function is to control
which principals (persons, processes, machines, . . .) have access to which
resources in the system — which files they can read, which programs they can
execute, how they share data with other principals, and so on.
Access control works at a number of levels (Figure 4.1).
Application
Middleware
Operating system
Hardware

Figure 4.1: Access controls at different levels in a system

93

94

Chapter 4

■

Access Control

1. The access control mechanisms the user sees at the application level may
express a very rich and complex security policy. A modern online business could assign staff to one of dozens of different roles, each of which
could initiate some subset of several hundred possible transactions in
the system. Some of these (such as refunds) might require dual control or
approval from a supervisor. And that’s nothing compared with the complexity of the access controls on a modern social networking site, which
will have a thicket of rules and options about who can see, copy, and
search what data from whom.
2. The applications may be written on top of middleware, such as a
database management system or bookkeeping package, which enforces
a number of protection properties. For example, bookkeeping software may ensure that a transaction which debits one ledger for a
certain amount must credit another ledger for the same amount, while
database software typically has access controls specifying which dictionaries a given user can select, and which procedures they can run.
3. The middleware will use facilities provided by the underlying operating
system. As this constructs resources such as files and communications
ports from lower level components, it acquires the responsibility for providing ways to control access to them.
4. Finally, the operating system access controls will usually rely on hardware features provided by the processor or by associated memory
management hardware. These control which memory addresses a given
process can access.
As we work up from the hardware through the operating system and
middleware to the application layer, the controls become progressively more
complex and less reliable. Most actual computer frauds involve staff accidentally discovering features of the application code that they can exploit in an
opportunistic way, or just abusing features of the application that they were
trusted not to. However, loopholes in access-control mechanisms — such as in
database software used in a large number of web servers — can expose many
systems simultaneously and compel large numbers of companies to patch or
rewrite their products. So in this chapter, we will focus on the fundamentals:
access control at the hardware, operating system and database levels. You
have to understand the basic principles to design serviceable application-level
controls too (I give many examples in Part II of how to combine access controls
with the needs of specific applications).
As with the other building blocks discussed so far, access control makes
sense only in the context of a protection goal, typically expressed as a security
policy. PCs carry an unfortunate legacy, in that the old single-user operating

4.1 Introduction

systems such as DOS and Win95/98 let any process modify any data; as a
result many applications won’t run unless they are run with administrator
privileges and thus given access to the whole machine. Insisting that your
software run as administrator is also more convenient for the programmer.
But people do have implicit protection goals; you don’t expect a shrinkwrap program to trash your hard disk. So an explicit security policy is a
good idea.
Now one of the biggest challenges in computer security is preventing one
program from interfering with another. You don’t want a virus to be able to
steal the passwords from your browser, or to patch a banking application so
as to steal your money. In addition, many everyday reliability problems stem
from applications interacting with each other or with the system configuration.
However, it’s difficult to separate applications when the customer wants to
share data. It would make phishing much harder if people were simply unable
to paste URLs from emails into a browser, but that would make everyday
life much harder too. The single-user history of personal computing has got
people used to all sorts of ways of working that are not really consistent
with separating applications and their data in useful ways. Indeed, one senior
manager at Microsoft took the view in 2000 that there was really nothing for
operating-system access controls to do, as client PCs were single-user and
server PCs were single-application.
The pendulum is now swinging back. Hosting centres make increasing
use of virtualization; having machines exclusively for your own use costs
several times what you pay for the same resource on shared machines. The
Trusted Computing initiative, and Microsoft Vista, place more emphasis on
separating applications from each other; even if you don’t care about security,
preventing your programs from overwriting each others’ configuration files
should make your PC much more reliable. More secure operating systems
have led to ever more technical attacks on software other than the operating
system; you don’t want your brokerage account hacked via a computer game
you downloaded that later turns out to be insecure. And employers would like
ways of ensuring that employees’ laptops don’t pick up malware at home, so
it makes sense to have one partition (or virtual machine) for work and another
for play. Undoing the damage done by many years of information-sharing
promiscuity will be hard, but in the medium term we might reasonably hope
for a framework that enables interactions between applications to be restricted
to controllable interfaces.
Many access control systems build on the mechanisms provided by the
operating system. I will start off by discussing operating-system protection
mechanisms that support the isolation of multiple processes. These came first
historically — being invented along with the first time-sharing systems in the

95

96

Chapter 4

■

Access Control

1960s — and they remain the foundation on which many higher-layer mechanisms are built. I will then go on to discuss database systems, which provide
broadly similar access control mechanisms that may or may not be tied to the
operating-systems mechanisms. Finally I’ll discuss three advanced protection
techniques — sandboxing, virtualization and ‘Trusted Computing’. Sandboxing
is an application-level control, run for example in a browser to restrict what
mobile code can do; virtualization runs underneath the operating system,
creating two or more independent virtual machines between which information flows can be controlled or prevented; and Trusted Computing is a
project to create two virtual machines side-by-side, one being the ‘old, insecure’ version of an operating system and the second being a more restricted
environment in which security-critical operations such as cryptography can
be carried out.
The latest Microsoft system, Vista, is trying to move away from running
all code with administrator privilege, and these three modern techniques are
each, independently, trying to achieve the same thing — to get us back where
we’d be if all applications had to run with user privileges rather than as the
administrator. That is more or less where computing was in the 1970s, when
people ran their code as unprivileged processes on time-shared minicomputers
and mainframes. Only time will tell whether we can recapture the lost Eden
of order and control alluded to in the quote from Roger Needham at the
start of this chapter, and to escape the messy reality of today to which Rick
Maybury’s quote refers; but certainly the attempt is worth making.

4.2

Operating System Access Controls

The access controls provided with an operating system typically authenticate
principals using a mechanism such as passwords or Kerberos, then mediate
their access to files, communications ports and other system resources.
Their effect can often be modelled by a matrix of access permissions, with
columns for files and rows for users. We’ll write r for permission to read, w
for permission to write, x for permission to execute a program, and - for no
access at all, as shown in Figure 4.2.
Operating Accounts Accounting Audit
System Program
Data
Trail
Sam
rwx
rwx
rw
r
Alice
x
x
rw
–
Bob
rx
r
r
r
Figure 4.2: Naive access control matrix

4.2 Operating System Access Controls

In this simplified example, Sam is the system administrator and has universal
access (except to the audit trail, which even he should only be able to read).
Alice, the manager, needs to execute the operating system and application,
but only through the approved interfaces — she mustn’t have the ability to
tamper with them. She also needs to read and write the data. Bob, the auditor,
can read everything.
This is often enough, but in the specific case of a bookkeeping system
it’s not quite what we need. We want to ensure that transactions are wellformed — that each debit is matched by a credit somewhere else — so we
would not want Alice to have uninhibited write access to the account file.
We would also rather that Sam didn’t have this access. So we would prefer
that write access to the accounting data file be possible only via the accounting
program. The access permissions might now look like in Figure 4.3:

User

Operating Accounts Accounting Audit
System Program
Data
Trail
Sam
rwx
rwx
r
r
Alice
rx
x
–
–
Accounts program
rx
r
rw
w
Bob
rx
r
r
r
Figure 4.3: Access control matrix for bookkeeping

Another way of expressing a policy of this type would be with access triples
of (user, program, file). In the general case, our concern isn’t with a program
so much as a protection domain which is a set of processes or threads which
share access to the same resources (though at any given time they might have
different files open or different scheduling priorities).
Access control matrices (whether in two or three dimensions) can be used to
implement protection mechanisms as well as just model them. But they do not
scale well. For instance, a bank with 50,000 staff and 300 applications would
have an access control matrix of 15,000,000 entries. This is inconveniently
large. It might not only impose a performance problem but also be vulnerable
to administrators’ mistakes. We will usually need a more compact way of
storing and managing this information. The two main ways of doing this are
to compress the users and to compress the rights. As for the first of these, the
simplest is to use groups or roles to manage the privileges of large sets of users
simultaneously, while in the second we may store the access control matrix
either by columns (access control lists) or rows (capabilities, sometimes known
as ‘tickets’) [1102, 1344]. (There are more complex approaches involving policy
engines, but let’s learn to walk before we try to run.)

97

98

Chapter 4

4.2.1

■

Access Control

Groups and Roles

When we look at large organisations, we usually find that most staff fit into
one or other of a small number of categories. A bank might have 40 or 50:
teller, chief teller, branch accountant, branch manager, and so on. Only a few
dozen people (security manager, chief foreign exchange dealer, . . .) will need
to have their access rights defined individually.
So we want a small number of pre-defined groups, or functional roles,
to which staff can be assigned. Some people use the words group and role
interchangeably, and with many systems they are; but the more careful
definition is that a group is a list of principals, while a role is a fixed set of
access permissions that one or more principals may assume for a period of time
using some defined procedure. The classic example of a role is the officer of the
watch on a ship. There is exactly one watchkeeper at any one time, and there
is a formal procedure whereby one officer relieves another when the watch
changes. In fact, in most government and business applications, it’s the role
that matters rather than the individual.
Groups and roles can be combined. The officers of the watch of all ships currently
at sea is a group of roles. In banking, the manager of the Cambridge branch
might have his or her privileges expressed by membership of the group
manager and assumption of the role acting manager of Cambridge branch. The
group manager might express a rank in the organisation (and perhaps even a
salary band) while the role acting manager might include an assistant accountant
standing in while the manager, deputy manager, and branch accountant are
all off sick.
Whether we need to be careful about this distinction is a matter for the
application. In a warship, we want even an ordinary seaman to be allowed to
stand watch if everyone more senior has been killed. In a bank, we might have
a policy that ‘transfers over $10m must be approved by two staff, one with
rank at least manager and one with rank at least assistant accountant’. If the
branch manager is sick, then the assistant accountant acting as manager might
have to get the regional head office to provide the second signature on a large
transfer.
Operating-system level support is available for groups and roles, but its
appearance has been fairly recent and its uptake is still slow. Developers
used to implement this kind of functionality in application code, or as custom
middleware (in the 1980s I worked on two bank projects where group support
was hand-coded as extensions to the mainframe operating system). Windows
2000 introduced extensive support for groups, while academic researchers
have done quite a lot of work since the mid-90s on role-based access control
(RBAC), which I’ll discuss further in Part II, and which is starting to be rolled
out in some large applications.

4.2 Operating System Access Controls

4.2.2 Access Control Lists
Another way of simplifying the management of access rights is to store the
access control matrix a column at a time, along with the resource to which
the column refers. This is called an access control list or ACL (pronounced
‘ackle’). In the first of our above examples, the ACL for file 3 (the account file)
might look as shown here in Figure 4.4.
ACLs have a number of advantages and disadvantages as a means of
managing security state. These can be divided into general properties of ACLs,
and specific properties of particular implementations.
ACLs are a natural choice in environments where users manage their
own file security, and became widespread in the Unix systems common in
universities and science labs from the 1970s. They are the basic access control
mechanism in Unix-based systems such as GNU/Linux and Apple’s OS/X;
the access controls in Windows are also based on ACLs, but have become
more complex over time. Where access control policy is set centrally, ACLs
are suited to environments where protection is data-oriented; they are less
suited where the user population is large and constantly changing, or where
users want to be able to delegate their authority to run a particular program
to another user for some set period of time. ACLs are simple to implement,
but are not efficient as a means of doing security checking at runtime, as the
typical operating system knows which user is running a particular program,
rather than what files it has been authorized to access since it was invoked.
The operating system must either check the ACL at each file access, or keep
track of the active access rights in some other way.
Finally, distributing the access rules into ACLs means that it can be tedious
to find all the files to which a user has access. Revoking the access of an
employee who has just been fired will usually have to be done by cancelling
their password or other authentication mechanism. It may also be tedious to
run system-wide checks; for example, verifying that no files have been left
world-writable could involve checking ACLs on millions of user files.
Let’s look at two important examples of ACLs — their implementation in
Unix and Windows.

User Accounting
Data
Sam
rw
Alice
rw
Bob
r
Figure 4.4: Access control list (ACL)

99

100

Chapter 4

4.2.3

■

Access Control

Unix Operating System Security

In Unix (including its popular variant Linux), files are not allowed to have
arbitrary access control lists, but simply rwx attributes for the resource owner,
the group, and the world. These attributes allow the file to be read, written
and executed. The access control list as normally displayed has a flag to show
whether the file is a directory, then flags r, w and x for world, group and owner
respectively; it then has the owner’s name and the group name. A directory
with all flags set would have the ACL:
drwxrwxrwx Alice Accounts

In our first example in Figure 4.3, the ACL of file 3 would be:
-rw-r- - - - - Alice Accounts

This records that the file is not a directory; the file owner can read and write
it; group members can read it but not write it; non-group members have no
access at all; the file owner is Alice; and the group is Accounts.
In Unix, the program that gets control when the machine is booted (the
operating system kernel) runs as the supervisor, and has unrestricted access
to the whole machine. All other programs run as users and have their access
mediated by the supervisor. Access decisions are made on the basis of the
userid associated with the program. However if this is zero (root), then
the access control decision is ‘yes’. So root can do what it likes — access any
file, become any user, or whatever. What’s more, there are certain things that
only root can do, such as starting certain communication processes. The root
userid is typically made available to the system administrator.
This means that (with most flavours of Unix) the system administrator can
do anything, so we have difficulty implementing an audit trail as a file that
he cannot modify. This not only means that, in our example, Sam could tinker
with the accounts, and have difficulty defending himself if he were falsely
accused of tinkering, but that a hacker who managed to become the system
administrator could remove all evidence of his intrusion.
The Berkeley distributions, including FreeBSD and OS/X, go some way
towards fixing the problem. Files can be set to be append-only, immutable or
undeletable for user, system or both. When set by a user at a sufficient security
level during the boot process, they cannot be overridden or removed later, even
by root. Various military variants go to even greater trouble to allow separation
of duty. However the simplest and most common way to protect logs against
root compromise is to keep them separate. In the old days that meant sending
the system log to a printer in a locked room; nowadays, given the volumes of
data, it means sending it to another machine, administered by somebody else.
Second, ACLs only contain the names of users, not of programs; so there
is no straightforward way to implement access triples of (user, program, file).

4.2 Operating System Access Controls

Instead, Unix provides an indirect method: the set-user-id (suid) file attribute.
The owner of a program can mark it as suid, which enables it to run with the
privilege of its owner rather than the privilege of the user who has invoked
it. So in order to achieve the functionality needed by our second example
above, we could create a user ‘account-package’ to own file 2 (the accounts
package), make the file suid and place it in a directory to which Alice has
access. This special user can then be given the access control attributes the
accounts program needs.
One way of looking at this is that an access control problem that is naturally
modelled in three dimensions — by triples (user, program, data) — is being
implemented using two-dimensional mechanisms. These mechanisms are
much less intuitive than triples and people make many mistakes implementing
them. Programmers are often lazy or facing tight deadlines; so they just make
the application suid root, so it can do anything. This practice leads to
some rather shocking security holes. The responsibility for making access
control decisions is moved from the operating system environment to the
program, and most programmers are insufficiently experienced and careful
to check everything they should. (It’s hard to know what to check, as the
person invoking a suid root program controls its environment and can often
manipulate this to cause protection failures.)
Third, ACLs are not very good at expressing mutable state. Suppose we
want a transaction to be authorised by a manager and an accountant before
it’s acted on; we can either do this at the application level (say, by having
queues of transactions awaiting a second signature) or by doing something
fancy with suid. In general, managing stateful access rules is difficult; this can
even complicate the revocation of users who have just been fired, as it can be
hard to track down the files they might have open.
Fourth, the Unix ACL only names one user. Older versions allow a process
to hold only one group id at a time and force it to use a privileged program
to access other groups; newer Unix systems put a process in all groups
that the user is in. This is still much less expressive than one might like. In
theory, the ACL and suid mechanisms can often be used to achieve the desired
effect. In practice, programmers often can’t be bothered to figure out how to
do this, and design their code to require much more privilege than it really
ought to have.

4.2.4 Apple’s OS/X
Apple’s OS/X operating system is based on the FreeBSD version of Unix running on top of the Mach kernel. The BSD layer provides memory protection;
applications cannot access system memory (or each others’) unless running
with advanced permissions. This means, for example, that you can kill a
wedged application using the ‘Force Quit’ command; you usually do not have

101

102

Chapter 4

■

Access Control

to reboot the system. On top of this Unix core are a number of graphics components, including OpenGL, Quartz, Quicktime and Carbon, while at the surface
the Aqua user interface provides an elegant and coherent view to the user.
At the file system level, OS/X is almost a standard Unix. The default
installation has the root account disabled, but users who may administer the
system are in a group ‘wheel’ that allows them to su to root. The most visible
implication is that if you are such a user, you can install programs (you are
asked for the root password when you do so). This may be a slightly better
approach than Windows (up till XP) or Linux, which in practice let only
administrators install software but do not insist on an authentication step
when they do so; the many Windows users who run as administrator for
convenience do dreadful things by mistake (and malware they download does
dreadful things deliberately). Although Microsoft is struggling to catch up
with Vista, as I’ll discuss below, Apple’s advantage may be increased further
by OS/X version 10.5 (Leopard), which is based on TrustedBSD, a variant of
BSD developed for government systems that incorporates mandatory access
control. (I’ll discuss this in Chapter 8.)

4.2.5

Windows – Basic Architecture

The most widespread PC operating system is Windows, whose protection
has been largely based on access control lists since Windows NT. The current
version of Windows (Vista) is fairly complex, so it’s helpful to trace its
antecedents. The protection in Windows NT (Windows v4) was very much
like Unix, and was inspired by it, and has since followed the Microsoft
philosophy of ‘embrace and extend’.
First, rather than just read, write and execute there are separate attributes for
take ownership, change permissions and delete, which means that more flexible
delegation can be supported. These attributes apply to groups as well as users,
and group permissions allow you to achieve much the same effect as suid
programs in Unix. Attributes are not simply on or off, as in Unix, but have
multiple values: you can set AccessDenied, AccessAllowed or SystemAudit. These
are parsed in that order. If an AccessDenied is encountered in an ACL for the
relevant user or group, then no access is permitted regardless of any conflicting
AccessAllowed flags. A benefit of the richer syntax is that you can arrange
matters so that everyday configuration tasks, such as installing printers, don’t
require full administrator privileges. (This is rarely done, though.)
Second, users and resources can be partitioned into domains with distinct
administrators, and trust can be inherited between domains in one direction
or both. In a typical large company, you might put all the users into a
personnel domain administered by Human Resources, while resources such
as servers and printers could be in resource domains under departmental
control; individual workstations may even be administered by their users.

4.2 Operating System Access Controls

Things would be arranged so that the departmental resource domains trust
the user domain, but not vice versa — so a corrupt or careless departmental
administrator can’t do much damage outside his own domain. The individual
workstations would in turn trust the department (but not vice versa) so that
users can perform tasks that require local privilege (installing many software
packages requires this). Administrators are still all-powerful (so you can’t
create truly tamper-resistant audit trails without using write-once storage
devices or writing to machines controlled by others) but the damage they can
do can be limited by suitable organisation. The data structure used to manage
all this, and hide the ACL details from the user interface, is called the Registry.
Problems with designing a Windows security architecture in very large
organisations include naming issues (which I’ll explore in Chapter 6), the way
domains scale as the number of principals increases (badly), and the restriction
that a user in another domain can’t be an administrator (which can cause
complex interactions between local and global groups).
One peculiarity of Windows is that everyone is a principal, not a default or an
absence of control, so ‘remove everyone’ means just stop a file being generally
accessible. A resource can be locked quickly by setting everyone to have no
access. This brings us naturally to the subject of capabilities.

4.2.6

Capabilities

The next way to manage the access control matrix is to store it by rows. These
are called capabilities, and in our example in Figure 4.2 above, Bob’s capabilities
would be as in Figure 4.5 here:
User Operating Accounts Accounting Audit
System Program
Data
Trail
Bob
rx
r
r
r
Figure 4.5: A capability

The strengths and weaknesses of capabilities are more or less the opposite
of ACLs. Runtime security checking is more efficient, and we can delegate a
right without much difficulty: Bob could create a certificate saying ‘Here is my
capability and I hereby delegate to David the right to read file 4 from 9am to
1pm, signed Bob’. On the other hand, changing a file’s status can suddenly
become more tricky as it can be difficult to find out which users have access.
This can be tiresome when we have to investigate an incident or prepare
evidence of a crime.
There were a number of experimental implementations in the 1970s, which
were rather like file passwords; users would get hard-to-guess bitstrings for
the various read, write and other capabilities to which they were entitled. It

103

104

Chapter 4

■

Access Control

was found that such an arrangement could give very comprehensive protection [1344]. It was not untypical to find that almost all of an operating system
could run in user mode rather than as supervisor, so operating system bugs
were not security critical. (In fact, many operating system bugs caused security
violations, which made debugging the operating system much easier.)
The IBM AS/400 series systems employed capability-based protection, and
enjoyed some commercial success. Now capabilities have made a limited
comeback in the form of public key certificates. I’ll discuss the mechanisms
of public key cryptography in Chapter 5, and give more concrete details of
certificate-based systems, such as SSL/TLS, in Part II. For now, think of a
public key certificate as a credential, signed by some authority, which declares
that the holder of a certain cryptographic key is a certain person, or a member
of some group, or the holder of some privilege.
As an example of where certificate-based capabilities can be useful, consider
a hospital. If we implemented a rule like ‘a nurse shall have access to all the
patients who are on her ward, or who have been there in the last 90 days’
naively, each access control decision in the patient record system will require
several references to administrative systems, to find out which nurses and
which patients were on which ward, when. So a failure of the administrative
systems can now affect patient safety much more directly than before, and
this is clearly a bad thing. Matters can be much simplified by giving nurses
certificates which entitle them to access the files associated with their current
ward. Such a system has been used for several years at our university hospital.
Public key certificates are often considered to be ‘crypto’ rather than ‘access
control’, with the result that their implications for access control policies
and architectures are not thought through. The lessons that could have been
learned from the capability systems of the 1970s are generally having to be
rediscovered (the hard way). In general, the boundary between crypto and
access control is a fault line where things can easily go wrong. The experts often
come from different backgrounds, and the products from different suppliers.

4.2.7

Windows – Added Features

A number of systems, from mainframe access control products to research
systems, have combined ACLs and capabilities in an attempt to get the best of
both worlds. But the most important application of capabilities is in Windows.
Windows 2000 added capabilities in two ways which can override or
complement the ACLs of Windows NT. First, users or groups can be either
whitelisted or blacklisted by means of profiles. (Some limited blacklisting was
also possible in NT4.) Security policy is set by groups rather than for the
system as a whole. Groups are intended to be the primary method for
centralized configuration management and control (group policy overrides
individual profiles). Group policy can be associated with sites, domains or

4.2 Operating System Access Controls

organizational units, so it can start to tackle some of the real complexity
problems with naming. Policies can be created using standard tools or custom
coded. Groups are defined in the Active Directory, an object-oriented database
that organises users, groups, machines, and organisational units within a
domain in a hierarchical namespace, indexing them so they can be searched
for on any attribute. There are also finer grained access control lists on
individual resources.
As already mentioned, Windows adopted Kerberos from Windows 2000
as its main means of authenticating users across networks1 . This is encapsulated behind the Security Support Provider Interface (SSPI) which enables
administrators to plug in other authentication services.
This brings us to the second way in which capabilities insinuate their way
into Windows: in many applications, people use the public key protocol TLS,
which is widely used on the web, and which is based on public key certificates.
The management of these certificates can provide another, capability-oriented,
layer of access control outside the purview of the Active Directory.
The latest version of Windows, Vista, introduces a further set of protection
mechanisms. Probably the most important is a package of measures aimed at
getting away from the previous default situation of all software running as root.
First, the kernel is closed off to developers; second, the graphics subsystem
is removed from the kernel, as are most drivers; and third, User Account
Control (UAC) replaces the default administrator privilege with user defaults
instead. This involved extensive changes; in XP, many routine tasks required
administrative privilege and this meant that enterprises usually made all their
users administrators, which made it difficult to contain the effects of malware.
Also, developers wrote their software on the assumption that it would have
access to all system resources.
In Vista, when an administrator logs on, she is given two access tokens: a
standard one and an admin one. The standard token is used to start the desktop,
explorer.exe, which acts as the parent process for later user processes. This
means, for example, that even administrators browse the web as normal users,
and malware they download can’t overwrite system files unless given later
authorisation. When a task is started that requires admin privilege, then a user
who has it gets an elevation prompt asking her to authorise it by entering an
admin password. (This brings Windows into line with Apple’s OS/X although
the details under the hood differ somewhat.)
1
It was in fact a proprietary variant, with changes to the ticket format which prevent Windows
clients from working with existing Unix Kerberos infrastructures. The documentation for the
changes was released on condition that it was not used to make compatible implementations.
Microsoft’s goal was to get everyone to install Win2K Kerberos servers. This caused an outcry in
the open systems community [121]. Since then, the European Union prosecuted an antitrust case
against Microsoft that resulted in interface specifications being made available in late 2006.

105

106

Chapter 4

■

Access Control

Of course, admin users are often tricked into installing malicious software,
and so Vista provides further controls in the form of file integrity levels. I’ll
discuss these along with other mandatory access controls in Chapter 8 but
the basic idea is that low-integrity processes (such as code you download
from the Internet) should not be able to modify high-integrity data (such
as system files). It remains to be seen how effective these measures will be;
home users will probably want to bypass them to get stuff to work, while
Microsoft is providing ever-more sophisticated tools to enable IT managers to
lock down corporate networks — to the point, for example, of preventing most
users from installing anything from removable media. UAC and mandatory
integrity controls can certainly play a role in this ecology, but we’ll have to
wait and see how things develop.
The final problem with which the Vista developers grappled is the fact that
large numbers of existing applications expect to run as root, so that they can
fool about with registry settings (for a hall of shame, see [579]). According to
the Microsoft folks, this is a major reason for Windows’ lack of robustness:
applications monkey with system resources in incompatible ways. So there is
an Application Information Service that launches applications which require
elevated privileges to run. Vista uses virtualization technology for legacy
applications: if they modify the registry, for example, they don’t modify the
‘real’ registry but simply the version of it that they can see. This is seen as a
‘short-term fix’ [885]. I expect it will be around for a long time, and I’m curious
to see whether the added complexity will be worth the reduced malware risk.
Despite virtualisation, the bugbear with Vista is compatibility. As this book
went to press in early January 2008, sales of Vista were still sluggish, with
personal users complaining that games and other applications just didn’t
work, while business users were waiting for service pack 1 and postponing
large-scale roll-out to late 2008 or even early 2009. It has clearly been expensive
for Microsoft to move away from running everything as root, but it’s clearly a
necessary move and they deserve full credit for biting the bullet.
To sum up, Windows provides a richer and more flexible set of access
control tools than any system previously sold in mass markets. It does still
have design limitations. Implementing roles whose requirements differ from
those of groups could be tricky in some applications; SSL certificates are the
obvious way to do this but require an external management infrastructure.
Second, Windows is still (in its consumer incarnations) a single-user operating
system in the sense that only one person can operate a PC at a time. Thus
if I want to run an unprivileged, sacrificial user on my PC for accessing
untrustworthy web sites that might contain malicious code, I have to log off
and log on again, or use other techniques which are so inconvenient that
few users will bother. (On my Mac, I can run two users simultaneously and
switch between them quickly.) So Vista should be seen as the latest step
on a journey, rather than a destination. The initial version also has some

4.2 Operating System Access Controls

undesirable implementation quirks. For example, it uses some odd heuristics
to try to maintain backwards compatibility with programs that assume they’ll
run as administrator: if I compile a C++ program called Fred Installer.exe
then Vista will ask for elevated privilege to run it, and tell it that it’s running
on Windows XP, while if I call the program simply Fred.exe it will run as
user and be told that it’s running on Vista [797]. Determining a program’s
privileges on the basis of its filename is just bizarre.
And finally, there are serious usability issues. For example, most users still
run administrator accounts all the time, and will be tempted to disable UAC;
if they don’t, they’ll become habituated to clicking away the UAC dialog box
that forever asks them if they really meant to do what they just tried to. For
these reasons, UAC may be much less effective in practice than it might be in
theory [555]. We will no doubt see in due course.
It’s interesting to think about what future access controls could support,
for example, an electronic banking application that would be protected from
malware running on the same machine. Microsoft did come up with some
ideas in the context of its ‘Trusted Computing’ project, which I’ll describe
below in section 4.2.11, but they didn’t make it into Vista.

4.2.8

Middleware

Doing access control at the level of files and programs was all very well in the
early days of computing, when these were the resources that mattered. Since
about the 1980s, growing scale and complexity has meant led to access control
being done at other levels instead of (sometimes as well as) at the operating
system level. For example, a bank’s branch bookkeeping system will typically
run on top of a database product, and the database looks to the operating
system as one large file. This means that the access control has to be done in
the database; all the operating system supplies it may be an authenticated ID
for each user who logs on.

4.2.8.1

Database Access Controls

Until the dotcom boom, database security was largely a back-room concern.
But it is now common for enterprises to have critical databases, that handle
inventory, dispatch and e-commerce, fronted by web servers that pass transactions to the databases directly. These databases now contain much of the
data of greatest relevance to our lives — such as bank accounts, vehicle registrations and employment records — and front-end failures sometimes expose
the database itself to random online users.
Database products, such as Oracle, DB2 and MySQL, have their own access
control mechanisms. As the database looks to the operating system as a single
large file, the most the operating system can do is to identify users and to

107

108

Chapter 4

■

Access Control

separate the database from other applications running on the same machine.
The database access controls are in general modelled on operating-system
mechanisms, with privileges typically available for both users and objects
(so the mechanisms are a mixture of access control lists and capabilities).
However, the typical database access control architecture is more complex
even than Windows: Oracle 10g has 173 system privileges of which at least six
can be used to take over the system completely [804]. There are more privileges
because a modern database product is very complex and some of the things
one wants to control involve much higher levels of abstraction than files or
processes. The flip side is that unless developers know what they’re doing,
they are likely to leave a back door open.
Some products let developers bypass operating-system controls. For
example, Oracle has both operating system accounts (whose users must be
authenticated externally by the platform) and database accounts (whose users
are authenticated directly by the Oracle software). It is often more convenient
to use database accounts as it saves the developer from synchronising his work
with the details of what other departments are doing. In many installations,
the database is accessible directly from the outside; this raises all sorts of
issues from default passwords to flaws in network protocols. Even where the
database is shielded from the outside by a web service front-end, this often
contains loopholes that let SQL code be inserted into the database.
Database security failures can also cause problems directly. The Slammer
worm in January 2003 propagated itself using a stack-overflow in Microsoft
SQL Server 2000 and created large amounts of traffic and compromised
machines sent large numbers of attack packets to random IP addresses.
Just as Windows is trickier to configure securely, because it’s more complex,
so the typical database system is trickier still, and it takes specialist knowledge
that’s beyond the scope of this book. Database security is now a discipline
in its own right; if you have to lock down a database system — or even just
review one as part of a broader assignment — I’d strongly recommend that
you read a specialist text, such as David Litchfield’s [804].

4.2.8.2

General Middleware Issues

There are a number of aspects common to middleware security and applicationlevel controls. The first is granularity: as the operating system works with files,
these are usually the smallest objects with which its access control mechanisms
can deal. The second is state. An access rule such as ‘a nurse can see the records
of any patient on her ward’ or ‘a transaction over $100,000 must be authorised
by a manager and an accountant’ both involve managing state: in the first
case the duty roster, and in the second the list of transactions that have so
far been authorised by only one principal. The third is level: we may end up
with separate access control systems at the machine, network and application

4.2 Operating System Access Controls

levels, and as these typically come from different vendors it may be difficult
to keep them consistent.
Ease of administration is often a critical bottleneck. In companies I’ve
advised, the administration of the operating system and the database system
have been done by different departments, which do not talk to each other;
and often user pressure drives IT departments to put in crude hacks which
make the various access control systems seem to work as one, but which open
up serious holes. An example is ‘single sign-on’. Despite the best efforts
of computer managers, most large companies accumulate systems of many
different architectures, so users get more and more logons to different systems
and the cost of administering them escalates. Many organisations want to give
each employee a single logon to all the machines on the network. Commercial
solutions may involve a single security server through which all logons must
pass, and the use of a smartcard to do multiple authentication protocols
for different systems. Such solutions are hard to engineer properly, and the
security of the best system can very easily be reduced to that of the worst.

4.2.8.3

ORBs and Policy Languages

These problems led researchers to look for ways in which access control for a
number of applications might be handled by standard middleware. Research
in the 1990s focussed on object request brokers (ORBs). An ORB is a software
component that mediates communications between objects (an object consists
of code and data bundled together, an abstraction used in object-oriented
languages such as C++). An ORB typically provides a means of controlling
calls that are made across protection domains. The Common Object Request
Broker Architecture (CORBA) is an attempt at an industry standard for objectoriented systems; a book on CORBA security is [182]. This technology is
starting to be adopted in some industries, such as telecomms.
Research since 2000 has included work on languages to express security policy, with projects such as XACML (Sun), XrML (ISO) and SecPAL (Microsoft).
They followed early work on ‘Policymaker’ by Matt Blaze and others [188],
and vary in their expressiveness. XrML deals with subjects and objects but
not relationships, so cannot easily express a concept such as ‘Alice is Bob’s
manager’. XACML does relationships but does not support universally quantified variables, so it cannot easily express ‘a child’s guardian may sign its
report card’ (which we might want to program as ‘if x is a child and y is x’s
guardian and z is x’s report card, then y may sign z). The initial interest in
these languages appears to come from the military and the rights-management
industry, both of which have relatively simple state in their access control policies. Indeed, DRM engineers have already developed a number of specialised
rights-management languages that are built into products such as Windows
Media Player and can express concepts such as ‘User X can play this file as

109

110

Chapter 4

■

Access Control

audio until the end of September and can burn it to a CD only once.’ The push
for interoperable DRM may create a demand for more general mechanisms
that can embrace and extend the products already in the field.
If a suitably expressive policy language emerges, and is adopted as a
standard scripting language on top of the access-control interfaces that major
applications provide to their administrators, it might provide some value by
enabling people to link up access controls when new services are constructed
on top of multiple existing services. There are perhaps two caveats. First,
people who implement access control when customizing a package are not
likely to do so as a full-time job, and so it may be better to let them use a
language with which they are familiar, in which they will be less likely to make
mistakes. Second, security composition is a hard problem; it’s easy to come up
with examples of systems that are secure in isolation but that break horribly
when joined up together. We’ll see many examples in Part II.
Finally, the higher in a system we build the protection mechanisms, the
more complex they’ll be, the more other software they’ll rely on, and the closer
they’ll be to the error-prone mark 1 human being — so the less dependable
they are likely to prove. Platform vendors such as Microsoft have more security
PhDs, and more experience in security design, than almost any application
vendor; and a customer who customises an application package usually has
less experience still. Code written by users is most likely to have glaring flaws.
For example, the fatal accidents that happened in healthcare as a result of the
Y2K bug were not platform failures, but errors in spreadsheets developed by
individual doctors, to do things like processing lab test results and calculating
radiology dosages. Letting random users write security-critical code carries
the same risks as letting them write safety-critical code.

4.2.9

Sandboxing and Proof-Carrying Code

The late 1990s saw the emergence of yet another way of implementing access
control: the software sandbox. This was introduced by Sun with its Java
programming language. The model is that a user wants to run some code
that she has downloaded from the web as an applet, but is concerned that
the applet might do something nasty, such as taking a list of all her files and
mailing it off to a software marketing company.
The designers of Java tackled this problem by providing a ‘sandbox’ for such
code — a restricted environment in which it has no access to the local hard
disk (or at most only temporary access to a restricted directory), and is only
allowed to communicate with the host it came from. These security objectives
are met by having the code executed by an interpreter — the Java Virtual
Machine (JVM) — which has only limited access rights [539]. Java is also used
on smartcards but (in current implementations at least) the JVM is in effect a
compiler external to the card, which raises the issue of how the code it outputs

4.2 Operating System Access Controls

can be got to the card in a trustworthy manner. Another application is in the
new Blu-ray format for high-definition DVDs; players have virtual machines
that execute rights-management code bundled with the disk. (I describe the
mechanisms in detail in section 22.2.6.2.)
An alternative is proof-carrying code. Here, code to be executed must carry
with it a proof that it doesn’t do anything that contravenes the local security
policy. This way, rather than using an interpreter with the resulting speed
penalty, one merely has to trust a short program that checks the proofs
supplied by downloaded programs before allowing them to be executed. The
overhead of a JVM is not necessary [956].
Both of these are less general alternatives to an architecture supporting
proper supervisor-level confinement.

4.2.10

Virtualization

This refers to systems that enable a single machine to emulate a number
of machines independently. It was invented in the 1960s by IBM [336]; back
when CPUs were very expensive, a single machine could be partitioned using
VM/370 into multiple virtual machines, so that a company that bought two
mainframes could use one for its production environment and the other as
a series of logically separate machines for development, testing, and minor
applications.
The move to PCs saw the emergence of virtual machine software for this
platform, with offerings from various vendors, notably VMware and (in opensource form) the Xen project. Virtualization is very attractive to the hosting
industry, as clients can be sold a share of a machine in a hosting centre for
much less than a whole machine. In the few years that robust products have
been available, their use has become extremely widespread.
At the client end, virtualization allows people to run a host operating system
on top of a guest (for example, Windows on top of Linux or OS/X) and this
offers not just flexibility but the prospect of better containment. For example,
an employee might have two copies of Windows running on his laptop — a
locked-down version with her office environment, and another for use at
home. The separation can be high-grade from the technical viewpoint; the
usual problem is operational. People may feel the need to share data between
the two virtual machines and resort to ad-hoc mechanisms, from USB sticks to
webmail accounts, that undermine the separation. Military system designers
are nonetheless very interested in virtualization; I discuss their uses of it in
section 8.5.3.

4.2.11 Trusted Computing
The ‘Trusted Computing’ initiative was launched by Microsoft, Intel, IBM, HP
and Compaq to provide a more secure PC. Their stated aim was to provide

111

112

Chapter 4

■

Access Control

software and hardware add-ons to the PC architecture that would enable
people to be sure that a given program was running on a machine with a
given specification; that is, that software had not been patched (whether by
the user or by other software) and was running on a identifiable type and
configuration of PC rather than on an emulator. The initial motivation was
to support digital rights management. The problem there was this: if Disney
was going to download a copy of a high-definition movie to Windows Media
Player on your PC, how could they be sure it wasn’t a hacked version, or
running on a copy of Windows that was itself running on top of Xen? In either
case, the movie might be ripped and end up on file-sharing systems.
The hardware proposal was to add a chip, the Trusted Platform Module
or TPM, which could monitor the PC at boot time and report its state to the
operating system; cryptographic keys would be made available depending on
this state. Thus if a platform were modified — for example, by changing the
boot ROM or the hard disk controller — different keys would be derived and
previously encrypted material would not be available. A PC would also be able
to use its TPM to certify to other PCs that it was in an ‘approved’ configuration,
a process called remote attestation. Of course, individual PCs might be hacked in
less detectable ways, such as by installing dual-ported memory or interfering
with the bus from the TPM to the CPU — but the idea was to exclude low-cost
break-once-run-anywhere attacks. Then again, the operating system will break
from time to time, and the media player; so the idea was to make the content
protection depend on as little as possible, and have revocation mechanisms
that would compel people to upgrade away from broken software.
Thus a vendor of digital content might only serve premium products to a
machine in an approved configuration. Furthermore, data-based access control
policies could be enforced. An example of these is found in the ‘Information
Rights Management’ mechanisms introduced with Office 2003; here, a file
can be marked with access controls in a rights expression language which can
state, for example, that it may only be read by certain named users and only
for a certain period of time. Word-processing documents (as well as movies)
could be marked ‘view three times only’; a drug dealer could arrange for
the spreadsheet with November’s cocaine shipments to be unreadable after
December, and so on.
There are objections to data-based access controls based on competition
policy, to which I’ll return in Part III. For now, my concern is the mechanisms.
The problem facing Microsoft was to maintain backwards compatibility with
the bad old world where thousands of buggy and insecure applications run as
administrator, while creating the possibility of new access domains to which
the old buggy code has no access. One proposed architecture, Palladium, was
unveiled in 2002; this envisaged running the old, insecure, software in parallel
with new, more secure components.

4.3 Hardware Protection

In addition to the normal operating system, Windows would have a ‘Nexus’,
a security kernel small enough to be verified, that would talk directly to the
TPM hardware and monitor what went on in the rest of the machine; and each
application would have a Nexus Control Program (NCP) that would run on
top of the Nexus in the secure virtual machine and manage critical things like
cryptographic keys. NCPs would have direct access to hardware. In this way,
a DRM program such as a media player could keep its crypto keys in its NCP
and use them to output content to the screen and speakers directly — so that
the plaintext content could not be stolen by spyware.
At the time of writing, the curtained memory features are not used in Vista;
presentations at Microsoft research workshops indicated that getting finegrained access control and virtualization to work at the middle layers of such
a complex operating system has turned out to be a massive task. Meanwhile
the TPM is available for secure storage of root keys for utilities such as hard
disk encryption; this is available as ’BitLocker’ in the more expensive versions
of Vista. It remains to be seen whether the more comprehensive vision of
Trusted Computing can be made to work; there’s a growing feeling in the
industry that it was too hard and, as it’s also politically toxic, it’s likely to be
quietly abandoned. Anyway, TPMs bring us to the more general problem of
the hardware protection mechanisms on which access controls are based.

4.3

Hardware Protection

Most access control systems set out not just to control what users can do, but
to limit what programs can do as well. In most systems, users can either write
programs, or download and install them. So programs may be buggy or even
malicious.
Preventing one process from interfering with another is the protection problem. The confinement problem is usually defined as that of preventing programs
communicating outward other than through authorized channels. There are
several flavours of each. The goal may be to prevent active interference, such
as memory overwriting, or to stop one process reading another’s memory
directly. This is what commercial operating systems set out to do. Military
systems may also try to protect metadata — data about other data, or subjects,
or processes — so that, for example, a user can’t find out what other users are
logged on to the system or what processes they’re running. In some applications, such as processing census data, confinement means allowing a program
to read data but not release anything about it other than the results of certain
constrained queries.
Unless one uses sandboxing techniques (which are too restrictive for general
programming environments), solving the confinement problem on a single
processor means, at the very least, having a mechanism that will stop one

113

114

Chapter 4

■

Access Control

program from overwriting another’s code or data. There may be areas of
memory that are shared in order to allow interprocess communication; but
programs must be protected from accidental or deliberate modification, and
they must have access to memory that is similarly protected.
This usually means that hardware access control must be integrated with the
processor’s memory management functions. A typical mechanism is segment
addressing. Memory is addressed by two registers, a segment register which
points to a segment of memory, and another address register which points to
a location within that segment. The segment registers are controlled by the
operating system, and often by a special component of it called the reference
monitor which links the access control mechanisms with the hardware.
The actual implementation has become more complex as the processors
themselves have. Early IBM mainframes had a two state CPU: the machine was
either in authorized state or it was not. In the latter case, the program was
restricted to a memory segment allocated by the operating system. In the
former, it could alter the segment registers at will. An authorized program
was one that was loaded from an authorized library.
Any desired access control policy can be implemented on top of this,
given suitable authorized libraries, but this is not always efficient; and system
security depends on keeping bad code (whether malicious or buggy) out of
the authorized libraries. So later processors offered more complex hardware
mechanisms. Multics, an operating system developed at MIT in the 1960’s and
which inspired the development of Unix, introduced rings of protection which
express differing levels of privilege: ring 0 programs had complete access to
disk, supervisor states ran in ring 2, and user code at various less privileged
levels [1139]. Its features have to some extent been adopted in more recent
processors, such as the Intel main processor line from the 80286 onwards.
There are a number of general problems with interfacing hardware and
software security mechanisms. For example, it often happens that a less
privileged process such as application code needs to invoke a more privileged
process such as a device driver. The mechanisms for doing this need to
be designed with some care, or security bugs can be expected. The IBM
mainframe operating system MVS, for example, had a bug in which a program
which executed a normal and an authorized task concurrently could make the
former authorized too [774]. Also, performance may depend quite drastically
on whether routines at different privilege levels are called by reference or by
value [1139].

4.3.1

Intel Processors, and ‘Trusted Computing’

Early Intel processors, such as the 8088/8086 used in early PCs, had no
distinction between system and user mode, and thus no protection at all — any
running program controlled the whole machine. The 80286 added protected

4.3 Hardware Protection

segment addressing and rings, so for the first time it could run proper operating
systems. The 80386 had built in virtual memory, and large enough memory
segments (4 Gb) that they could be ignored and the machine treated as a
32-bit flat address machine. The 486 and Pentium series chips added more
performance (caches, out of order execution and MMX).
The rings of protection are supported by a number of mechanisms. The
current privilege level can only be changed by a process in ring 0 (the kernel).
Procedures cannot access objects in lower level rings directly but there are
gates which allow execution of code at a different privilege level and which
manage the supporting infrastructure, such as multiple stack segments for
different privilege levels and exception handling. For more details, see [646].
The Pentium 3 finally added a new security feature — a processor serial
number. This caused a storm of protest because privacy advocates feared it
could be used for all sorts of ‘big brother’ purposes, which may have been
irrational as computers have all sorts of unique numbers in them that software
can use to tell which machine it’s running on (examples range from MAC
addresses to the serial numbers of hard disk controllers). At the time the serial
number was launched, Intel had planned to introduce cryptographic support
into the Pentium by 2000 in order to support DRM. Their thinking was that
as they earned 90% of their profits from PC microprocessors, where they had
90% of the market, they could only grow their business by expanding the
market for PCs; and since the business market was saturated, that meant sales
to homes where, it was thought, DRM would be a requirement.
Anyway, the outcry against the Pentium serial number led Intel to set up
an industry alliance, now called the Trusted Computing Group, to introduce
cryptography into the PC platform by means of a separate processor, the
Trusted Platform Module (TPM), which is a smartcard chip mounted on
the PC motherboard. The TPM works together with curtained memory features
introduced in the Pentium to enable operating system vendors to create
memory spaces isolated from each other, and even against a process in
one memory space running with administrator privileges. The mechanisms
proposed by Microsoft are described above, and have not been made available
in commercial releases of Windows at the time of writing.
One Intel hardware feature that has been implemented and used is the
x86 virtualization support, known as Intel VT (or its development name,
Vanderpool). AMD has an equivalent offering. Processor architectures such
as S/370 and M68000 are easy to virtualize, and the theoretical requirements
for this have been known for many years [1033]. The native Intel instruction
set, however, had instructions that were hard to virtualize, requiring messy
workarounds, such as patches to hosted operating systems. Processors with
these extensions can use products such as Xen to run unmodified copies of
guest operating systems. (It does appear, though, that if the Trusted Computing

115

116

Chapter 4

■

Access Control

mechanisms are ever implemented, it will be complex to make them work
alongside virtualization.)

4.3.2

ARM Processors

The ARM is the 32-bit processor core most commonly licensed to third party
vendors of embedded systems; hundreds of millions are used in mobile phones
and other consumer electronic devices. The original ARM (which stood for
Acorn Risc Machine) was the first commercial RISC design. ARM chips are
also used in many security products, from the Capstone chips used by the
US government to protect secret data, to the crypto accelerator boards from
firms like nCipher that do cryptography for large web sites. A fast multiplyand-accumulate instruction and low power consumption made the ARM very
attractive for embedded applications doing crypto and/or signal processing.
The standard reference is [508].
The ARM is licensed as a processor core, which chip designers can include
in their products, plus a number of optional add-ons. The basic core contains
separate banks of registers for user and system processes, plus a software
interrupt mechanism that puts the processor in supervisor mode and transfers
control to a process at a fixed address. The core contains no memory management, so ARM-based designs can have their hardware protection extensively
customized. A system control coprocessor is available to help with this. It can
support domains of processes that have similar access rights (and thus share
the same translation tables) but that retain some protection from each other.
This gives fast context switching. Standard product ARM CPU chips, from the
model 600 onwards, have this memory support built in.
There is a version, the Amulet, which uses self-timed logic. Eliminating
the clock saved power and reduces RF interference, but made it necessary to
introduce hardware protection features, such as register locking, into the main
processor itself so that contention between different hardware processes could
be managed. This is an interesting example of protection techniques typical of
an operating system being recycled in main-line processor design.

4.3.3

Security Processors

Specialist security processors range from the chips in smartcards, through the
TPM chips now fixed to most PC motherboards (which are basically smartcard
chips with parallel interfaces) and crypto accelerator boards, to specialist
crypto devices.
Many of the lower-cost smartcards still have 8-bit processors. Some of them
have memory management routines that let certain addresses be read only
when passwords are entered into a register in the preceding few instructions.
The goal was that the various principals with a stake in the card — perhaps

4.4 What Goes Wrong

a card manufacturer, an OEM, a network and a bank — can all have their
secrets on the card and yet be protected from each other. This may be a matter
of software; but some cards have small, hardwired access control matrices to
enforce this protection.
Many of the encryption devices used in banking to handle ATM PINs have
a further layer of application-level access control in the form of an ‘authorized
state’ which must be set (by two console passwords, or a physical key) when
PINs are to be printed. This is reminiscent of the old IBM mainframes, but is
used for manual rather than programmatic control: it enables a shift supervisor
to ensure that he is present when this job is run. Similar devices are used by
the military to distribute keys. I’ll discuss cryptoprocessors in more detail in
Chapter 16.

4.4

What Goes Wrong

Popular operating systems such as Unix / Linux and Windows are very large
and complex, so they have many bugs. They are used in a huge range of
systems, so their features are tested daily by millions of users under very
diverse circumstances. Consequently, many bugs are found and reported.
Thanks to the net, knowledge spreads widely and rapidly. Thus at any one
time, there may be dozens of security flaws that are known, and for which
attack scripts are circulating on the net. A vulnerability has a typical lifecycle
whereby it is discovered; reported to CERT or to the vendor; a patch is
shipped; the patch is reverse-engineered, and an exploit is produced for the
vulnerability; and people who did not apply the patch in time find that their
machines have been recruited to a botnet when their ISP cuts them off for
sending spam. There is a variant in which the vulnerability is exploited at
once rather than reported — often called a zero-day exploit as attacks happen
from day zero of the vulnerability’s known existence. The economics, and
the ecology, of the vulnerability lifecycle are the subject of intensive study by
security economists; I’ll discuss their findings in Part III.
The traditional goal of an attacker was to get a normal account on the system
and then become the system administrator, so he could take over the system
completely. The first step might have involved guessing, or social-engineering,
a password, and then using one of the many known operating-system bugs
that allow the transition from user to root. A taxonomy of such technical
flaws was compiled in 1993 by Carl Landwehr [774]. These involved failures
in the technical implementation, of which I will give examples in the next two
sections, and also in the higher level design; for example, the user interface
might induce people to mismanage access rights or do other stupid things
which cause the access control to be bypassed. I will give some examples in
section 4.4.3 below.

117

118

Chapter 4

■

Access Control

The user/root distinction has become less important in the last few years for
two reasons. First, Windows PCs have predominated, running applications
that insist on being run as administrator, so any application that can be
compromised gives administrator access. Second, attackers have come to
focus on compromising large numbers of PCs, which they can organise into
a botnet in order to send spam or phish and thus make money. Even if your
mail client were not running as administrator, it would still be useful to a
spammer who could control it. However, botnet herders tend to install rootkits
which, as their name suggests, run as root; and the user/root distinction does
still matter in business environments, where you do not want a compromised
web server or database application to expose your other applications as well.
Perhaps if large numbers of ordinary users start running Vista with User
Account Control enabled, it will make the botnet herders’ lives a bit harder.
We may at least hope.
In any case, the basic types of technical attack have not changed hugely
since the early 1990s and I’ll now consider them briefly.

4.4.1

Smashing the Stack

About half of the technical attacks on operating systems that are reported in
Computer Emergency Response Team (CERT) bulletins and security mailing lists
involve memory overwriting attacks, colloquially known as ‘smashing the
stack’. The proportion was even higher in the late 1990s and early 2000s but is
now dropping slowly.
The basic idea behind the stack-smashing attack is that programmers are
often careless about checking the size of arguments, so an attacker who passes
a long argument to a program may find that some of it gets treated as code
rather than data. A classic example was a vulnerability in the Unix finger
command. A widespread implementation of this would accept an argument
of any length, although only 256 bytes had been allocated for this argument
by the program. When an attacker used the command with a longer argument,
the trailing bytes of the argument ended up being executed by the system.
The usual technique is to arrange for the trailing bytes of the argument to
have a landing pad — a long space of no-operation (NOP) commands, or other
register commands that don’t change the control flow, and whose task is to
catch the processor if it executes any of them. The landing pad delivers the
processor to the attack code which will do something like creating a root
account with no password, or starting a shell with administrative privilege
directly (see Figure 4.6).
Many of the vulnerabilities reported routinely by CERT and bugtraq are
variants on this theme. I wrote in the first edition of this book, in 2001, ‘There
is really no excuse for the problem to continue, as it’s been well known for a
generation’. Yet it remains a problem.

4.4 What Goes Wrong

Malicious code

Malicious
argument

‘Landing pad’

Over-long
input

Target machine
memory map
Area allocated for
input buffer

Executable
code

Figure 4.6: Stack smashing attack

Most of the early 1960’s time sharing systems suffered from it, and fixed
it [549]. Penetration analysis efforts at the System Development Corporation
in the early ’70s showed that the problem of ‘unexpected parameters’ was still
one of the most frequently used attack strategies [799]. Intel’s 80286 processor introduced explicit parameter checking instructions — verify read, verify
write, and verify length — in 1982, but they were avoided by most software
designers to prevent architecture dependencies. In 1988, large numbers of
Unix computers were brought down simultaneously by the ‘Internet worm’,
which used the finger vulnerability described above, and thus brought memory overwriting attacks to the notice of the mass media [1206]. A 1998 survey
paper described memory overwriting attacks as the ‘attack of the decade’ [329].
Yet programmers still don’t check the size of arguments, and holes keep on
being found. The attack isn’t even limited to networked computer systems: at
least one smartcard could be defeated by passing it a message longer than its
programmer had anticipated.

4.4.2 Other Technical Attacks
In 2002, Microsoft announced a security initiative that involved every programmer being trained in how to write secure code. (The book they produced
for this, ‘Writing Secure Code’ by Michael Howard and David LeBlanc, is good;
I recommend it to my students [627].) Other tech companies have launched
similar training programmes. Despite the training and the tools, memory
overwriting attacks are still appearing, to the great frustration of software
company managers. However, they are perhaps half of all new vulnerabilities
now rather than the 90% they were in 2001.
The other new vulnerabilities are mostly variations on the same general
theme, in that they occur when data in grammar A is interpreted as being in
grammar B. A stack overflow is when data are accepted as input (e.g. a URL)

119

120

Chapter 4

■

Access Control

and end up being executed as machine code. They are essentially failures of
type safety.
A format string vulnerability arises when a machine accepts input data
as a formatting instruction (e.g. %n in the C command printf()). These
commonly arise when a programmer tries to print user-supplied data and
carelessly allows the print command to interpret any formatting instructions
in the string; this may allow the string’s author to write to the stack. There
are many other variants on the theme; buffer overflows can be induced by
improper string termination, passing an inadequately sized buffer to a path
manipulation function, and many other subtle errors. See Gary McGraw’s
book ‘Software Security’ [858] for a taxonomy.
SQL insertion attacks commonly arise when a careless web developer
passes user input to a back-end database without checking to see whether it
contains SQL code. The game is often given away by error messages, from
which a capable and motivated user may infer enough to mount an attack.
(Indeed, a survey of business websites in 2006 showed that over 10% were
potentially vulnerable [1234].) There are similar command-injection problems
afflicting other languages used by web developers, such as PHP and perl. The
remedy in general is to treat all user input as suspicious and validate it.
Checking data sizes is all very well when you get the buffer size calculation
correct, but when you make a mistake — for example, if you fail to consider
all the edge cases — you can end up with another type of attack called an
integer manipulation attack. Here, an overflow, underflow, wrap-around or
truncation can result in the ‘security’ code writing an inappropriate number
of bytes to the stack.
Once such type-safety attacks are dealt with, race conditions are probably
next. These occur when a transaction is carried out in two or more stages, and
it is possible for someone to alter it after the stage which involves verifying
access rights. I mentioned in Chapter 2 how a race condition can allow users
to log in as other users if the userid can be overwritten while the password
validation is in progress. Another classic example arose in the Unix command
to create a directory, ‘mkdir’, which used to work in two steps: the storage
was allocated, and then ownership was transferred to the user. Since these
steps were separate, a user could initiate a ‘mkdir’ in background, and if this
completed only the first step before being suspended, a second process could
be used to replace the newly created directory with a link to the password
file. Then the original process would resume, and change ownership of the
password file to the user. The /tmp directory, used for temporary files, can
often be abused in this way; the trick is to wait until an application run by a
privileged user writes a file here, then change it to a symbolic link to another
file somewhere else — which will be removed when the privileged user’s
application tries to delete the temporary file.

4.4 What Goes Wrong

A wide variety of other bugs have enabled users to assume root status
and take over the system. For example, the PDP-10 TENEX operating system
had the bug that the program address could overflow into the next bit of
the process state word which was the privilege-mode bit; this meant that a
program overflow could put a program in supervisor state. In another example,
some Unix implementations had the feature that if a user tried to execute the
command su when the maximum number of files were open, then su was
unable to open the password file and responded by giving the user root status.
In more modern systems, the most intractable user-to-root problems tend to be
feature interactions. For example, we’ve struggled with backup and recovery
systems. It’s convenient if you can let users recover their own files, rather than
having to call a sysadmin — but how do you protect information assets from
a time traveller, especially if the recovery system allows him to compose parts
of pathnames to get access to directories that were always closed to him? And
what if the recovery functionality is buried in an application to which he needs
access in order to do his job, and can be manipulated to give root access?
There have also been many bugs that allowed service denial attacks. For
example, Multics had a global limit on the number of files that could be open at
once, but no local limits. So a user could exhaust this limit and lock the system
so that not even the administrator could log on [774]. And until the late 1990’s,
most implementations of the Internet protocols allocated a fixed amount of
buffer space to process the SYN packets with which TCP/IP connections are
initiated. The result was SYN flooding attacks: by sending a large number of
SYN packets, an attacker could exhaust the available buffer space and prevent
the machine accepting any new connections. This is now fixed using syncookies,
which I’ll discuss in Part II.
The most recently discovered family of attacks of this kind are on system
call wrappers. These are software products that modify software behaviour
by intercepting the system calls it makes and performing some filtering or
manipulation. Some wrapper products do virtualization; others provide security extensions to operating systems. However Robert Watson has discovered
that such products may have synchronization bugs and race conditions that
allow an attacker to become root [1325]. (I’ll describe these in more detail in
section 18.3.) The proliferation of concurrency mechanisms everywhere, with
multiprocessor machines suddenly becoming the norm after many years in
which they were a research curiosity, may lead to race conditions being the
next big family of attacks.

4.4.3 User Interface Failures
One of the earliest attacks to be devised was the Trojan Horse, a program that
the administrator is invited to run and which will do some harm if he does
so. People would write games which checked occasionally whether the player

121

122

Chapter 4

■

Access Control

was the system administrator, and if so would create another administrator
account with a known password.
Another trick is to write a program that has the same name as a commonly
used system utility, such as the ls command which lists all the files in a Unix
directory, and design it to abuse the administrator privilege (if any) before
invoking the genuine utility. The next step is to complain to the administrator
that something is wrong with this directory. When the administrator enters
the directory and types ls to see what’s there, the damage is done. The fix
is simple: an administrator’s ‘PATH’ variable (the list of directories which
will be searched for a suitably named program when a command is invoked)
should not contain ‘.’ (the symbol for the current directory). Recent Unix
versions are shipped with this as a default; but it’s still an unnecessary trap
for the unwary.
Perhaps the most serious example of user interface failure, in terms of
the number of systems at risk, is in Windows. I refer to the fact that, until
Vista came along, a user needed to be the system administrator to install
anything2 . In theory this might have helped a bank preventing its branch
staff from running games on their PCs at lunchtime and picking up viruses.
But most environments are much less controlled, and many people need
to be able to install software to get their work done. So millions of people
have administrator privileges who shouldn’t need them, and are vulnerable
to attacks in which malicious code simply pops up a box telling them to
do something. Thank goodness Vista is moving away from this, but UAC
provides no protection where applications such as web servers must run as
root, are visible to the outside world, and contain software bugs that enable
them to be taken over.
Another example, which might be argued is an interface failure, comes
from the use of active content of various kinds. These can be a menace
because users have no intuitively clear way of controlling them. Javascript and
ActiveX in web pages, macros in Office documents and executables in email
attachments have all been used to launch serious attacks. Even Java, for all
its supposed security, has suffered a number of attacks that exploited careless
implementations [360]. However, many people (and many companies) are
unwilling to forego the bells and whistles which active content can provide,
and we saw in Chapter 2 how the marketing folks usually beat the security
folks (even in applications like banking).

4.4.4

Why So Many Things Go Wrong

We’ve already mentioned the basic problem faced by operating system security
designers: their products are huge and therefore buggy, and are tested by large
2

In theory a member of the Power Users Group in XP could but that made little difference.

4.4 What Goes Wrong

numbers of users in parallel, some of whom will publicize their discoveries
rather than reporting them to the vendor. Even if all bugs were reported
responsibly, this wouldn’t make much difference; almost all of the widely
exploited vulnerabilities over the last few years had already been patched.
(Indeed, Microsoft’s ‘Patch Tuesday’ each month is followed by ‘Exploit
Wednesday’ as the malware writers reverse the new vulnerabilities and attack
them before everyone’s patched them.)
There are other structural problems too. One of the more serious causes of
failure is kernel bloat. Under Unix, all device drivers, filesystems etc. must be
in the kernel. Until Vista, the Windows kernel used to contain drivers for a
large number of smartcards, card readers and the like, many of which were
written by the equipment vendors. So large quantities of code were trusted,
in that they are put inside the security perimeter. Some other systems, such as
MVS, introduced mechanisms that decrease the level of trust needed by many
utilities. However the means to do this in the most common operating systems
are few and relatively nonstandard.
Even more seriously, most application developers make their programs
run as root. The reasons for this are economic rather than technical, and are
easy enough to understand. A company trying to build market share for a
platform, such as an operating system, must appeal to its complementers — its
application developers — as well as to its users. It is easier for developers
if programs can run as root, so early Windows products allowed just that.
Once the vendor has a dominant position, the business logic is to increase the
security, and also to customise it so as to lock in both application developers
and users more tightly. This is now happening with Windows Vista as the
access control mechanisms become ever more complex, and different from
Linux and OS/X. A similar pattern, or too little security in the early stages of a
platform lifecycle and too much (of the wrong kind) later, has been observed
in other platforms from mainframes to mobile phones.
Making many applications and utilities run as root has repeatedly introduced horrible vulnerabilities where more limited privilege could have been
used with only a modicum of thought and a minor redesign. There are many
systems such as lpr/lpd — the Unix lineprinter subsystem — which does not
need to run as root but does anyway on most systems. This has also been a
source of security failures in the past (e.g., getting the printer to spool to the
password file).
Some applications need a certain amount of privilege. For example, mail
delivery agents must be able to deal with user mailboxes. But while a prudent
designer would restrict this privilege to a small part of the application, most
agents are written so that the whole program needs to run as root. The classic
example is sendmail, which has a long history of serious security holes; but
many other MTAs also have problems. The general effect is that a bug which

123

124

Chapter 4

■

Access Control

ought to compromise only one person’s mail may end up giving root privilege
to an outside attacker.
So we’re going to have some interesting times as developers come to grips
with UAC. The precedents are not all encouraging. Some programmers historically avoided the difficulty of getting non-root software installed and working
securely by simply leaving important shared data structures and resources
accessible to all users. Many old systems stored mail in a file per user in
a world-writeable directory, which makes mail forgery easy. The Unix file
utmp — the list of users logged in — was frequently used for security checking
of various kinds, but is also frequently world-writeable! This should have
been built as a service rather than a file — but fixing problems like these once
the initial design decisions have been made can be hard. I expect to see all
the old problems of 1970s multiuser systems come back again, as the complexity of using the Vista mechanisms properly just defeats many programmers
who aren’t security specialists and are just desparate to get something sort of
working so they can end the assignment, collect their bonus and move on.

4.4.5

Remedies

Some classes of vulnerability can be fixed using automatic tools. Stack overwriting attacks, for example, are largely due to the lack of proper bounds
checking in C (the language most operating systems are written in). There
are various tools (including free tools) available for checking C programs for
potential problems, and there is even a compiler patch called StackGuard
which puts a canary next to the return address on the stack. This can be a
random 32-bit value chosen when the program is started, and checked when a
function is torn down. If the stack has been overwritten meanwhile, then with
high probability the canary will change [329]. The availability of these tools,
and training initiatives such as Microsoft’s, have slowly reduced the number
of stack overflow errors. However, attack tools also improve, and attackers are
now finding bugs such as format string vulnerabilities and integer overflows
to which no-one paid much attention in the 1990s.
In general, much more effort needs to be put into design, coding and testing.
Architecture matters; having clean interfaces that evolve in a controlled way,
under the eagle eye of someone experienced who has a long-term stake in the
security of the product, can make a huge difference. (I’ll discuss this at greater
length in Part III.) Programs should only have as much privilege as they need:
the principle of least privilege [1102]. Software should also be designed so that
the default configuration, and in general, the easiest way of doing something,
should be safe. Sound architecture is critical in achieving safe defaults and
using least privilege. However, many systems are shipped with dangerous
defaults and messy code that potentially exposes all sorts of interfaces to
attacks like SQL injection that just shouldn’t happen.

4.4 What Goes Wrong

4.4.6 Environmental Creep
I have pointed out repeatedly that many security failures result from environmental change undermining a security model. Mechanisms that were adequate
in a restricted environment often fail in a more general one.
Access control mechanisms are no exception. Unix, for example, was originally designed as a ‘single user Multics’ (hence the name). It then became an
operating system to be used by a number of skilled and trustworthy people
in a laboratory who were sharing a single machine. In this environment the
function of the security mechanisms is mostly to contain mistakes; to prevent
one user’s typing errors or program crashes from deleting or overwriting
another user’s files. The original security mechanisms were quite adequate for
this purpose.
But Unix security became a classic ‘success disaster’. Unix was repeatedly
extended without proper consideration being given to how the protection
mechanisms also needed to be extended. The Berkeley versions assumed an
extension from a single machine to a network of machines that were all on
one LAN and all under one management. Mechanisms such as rhosts were
based on a tuple (username,hostname) rather than just a username, and saw the
beginning of the transfer of trust.
The Internet mechanisms (telnet, ftp, DNS, SMTP), which grew out of
Arpanet in the 1970’s, were written for mainframes on what was originally
a secure WAN. Mainframes were autonomous, the network was outside the
security protocols, and there was no transfer of authorization. Thus remote
authentication, which the Berkeley model was starting to make prudent, was
simply not supported. The Sun contributions (NFS, NIS, RPC etc.) were based
on a workstation model of the universe, with a multiple LAN environment
with distributed management but still usually in a single organisation. (A
proper tutorial on topics such as DNS and NFS is beyond the scope of this
book, but there is some more detailed background material in the section on
Vulnerabilities in Network Protocols in Chapter 21.)
Mixing all these different models of computation together has resulted in
chaos. Some of their initial assumptions still apply partially, but none of them
apply globally any more. The Internet now has hundreds of millions of PCs,
millions of LANs, thousands of interconnected WANs, and managements
which are not just independent but may be in conflict (including nation
states and substate groups that are at war with each other). Many PCs have
no management at all, and there’s a growing number of network-connected
Windows and Linux boxes in the form of fax machines, routers and other
embedded products that don’t ever get patched.
Users, instead of being trustworthy but occasionally incompetent, are now
largely incompetent — but some are both competent and hostile. Code used
to be simply buggy — but now there is a significant amount of malicious

125

126

Chapter 4

■

Access Control

code out there. Attacks on communications networks used to be the purview
of national intelligence agencies — now they can be done by script kiddies,
relatively unskilled people who have downloaded attack tools from the net
and launched them without any real idea of how they work.
So Unix and Internet security gives us yet another example of a system
that started out reasonably well designed but which was undermined by a
changing environment.
Windows Vista and its predecessors in the NT product series have more
extensive protection mechanisms than Unix, but have been around for much
less time. The same goes for database products such as Oracle. Realistically,
all we can say is that the jury is still out.

4.5

Summary

Access control mechanisms operate at a number of levels in a system, from
applications down through middleware to the operating system and the
hardware. Higher level mechanisms can be more expressive, but also tend to
be more vulnerable to attack for a variety of reasons ranging from intrinsic
complexity to implementer skill levels. Most attacks involve the opportunistic
exploitation of bugs, and software products that are very large, very widely
used, or both (as with operating systems and databases) are particularly likely
to have security bugs found and publicized. Systems at all levels are also
vulnerable to environmental changes which undermine the assumptions used
in their design.
The main function of access control is to limit the damage that can be
done by particular groups, users, and programs whether through error or
malice. The most important fielded examples are Unix and Windows, which
are similar in many respects, though Windows is more expressive. Database
products are often more expressive still (and thus even harder to implement
securely.) Access control is also an important part of the design of special
purpose hardware such as smartcards and other encryption devices. New
techniques are being developed to push back on the number of implementation
errors, such as stack overflow attacks; but new attacks are being found
continually, and the overall dependability of large software systems improves
only slowly.
The general concepts of access control from read, write and execute permissions to groups and roles will crop up again and again. In some distributed
systems, they may not be immediately obvious as the underlying mechanisms
can be quite different. An example comes from public key infrastructures,
which are a reimplementation of an old access control concept, the capability.
However, the basic mechanisms (and their problems) are pervasive.

Further Reading

Research Problems
Most of the issues in access control were identified by the 1960’s or early 1970’s
and were worked out on experimental systems such as Multics [1139] and the
CAP [1344]. Much of the research in access control systems since has involved
reworking the basic themes in new contexts, such as object oriented systems
and mobile code.
Recent threads of research include how to combine access control with the
admission control mechanisms used to provide quality of service guaranteed
in multimedia operating systems, and how to implement and manage access
control efficiently in large complex systems, using roles and policy languages.
However the largest single topic of research during 2003–6 has been ‘Trusted
Computing’, and how various trust mechanisms could be layered on top of
the mechanisms proposed by the Trusted Computing Group. The failure of
Windows Vista, as released in January 2007, to support remote attestation has
somewhat taken the wind out of the sails of this effort.
I suspect that a useful research topic for the next few years will be how
to engineer access control mechanisms that are not just robust but also
usable — by both programmers and end users. Separation is easy enough in
principle; one can have different machines, or different virtual machines, for
different tasks. But how happy would people be with an electronic banking
application that was so well walled off from the rest of the digital world that
they could not export figures from their bank statement into a spreadsheet?
I’ll discuss this problem at greater length when we come to mandatory access
controls in Chapter 8.

Further Reading
The best textbook to go to for a more detailed introduction to access control
issues is Dieter Gollmann’s ‘Computer Security’ [537]. A technical report from
Carl Landwehr gives a useful reference to many of the flaws found in
operating systems over the last 30 years or so [774]. One of the earliest
reports on the subject (and indeed on computer security in general) is by
Willis Ware [1319]. One of the most influential early papers is by Jerry Saltzer
and Mike Schroeder [1102], while Butler Lampson’s influential paper on the
confinement problem is at [768].
The classic description of Unix security is in the paper by Fred Grampp and
Robert Morris [550]. The most comprehensive textbook on this subject is Simson Garfinkel and Eugene Spafford’s Practical Unix and Internet Security [517],
while the classic on the Internet side of things is Bill Cheswick and Steve

127

128

Chapter 4

■

Access Control

Bellovin’s Firewalls and Internet Security [157], with many examples of network
attacks on Unix systems.
The protection mechanisms of Windows are described briefly in Gollmann.
For more detail, see the Microsoft online documentation; no doubt a number
of textbooks on Vista will appear soon. There is a history of microprocessor
architectures at [128], and a reference book for Java security written by its
architect Li Gong [539].
The field of software security is fast moving; the attacks that are catching
the headlines change significantly (at least in their details) from one year to the
next. The best recent book I’ve read is Gary McGraw’s [858]. But to keep up,
you should not just read textbooks, but follow the latest notices from CERT and
mailing lists such as bugtraq and books about the dark side such as Markus
Jakobsson and Zulfikar Ramzan’s [660].

CHAPTER

5

Cryptography
ZHQM ZMGM ZMFM
— G Julius Caesar
KXJEY UREBE ZWEHE WRYTU HEYFS KREHE GOYFI WTTTU OLKSY CAJPO BOTEI
ZONTX BYBWT GONEY CUZWR GDSON SXBOU YWRHE BAAHY USEDQ
— John F Kennedy

5.1

Introduction

Cryptography is where security engineering meets mathematics. It provides
us with the tools that underlie most modern security protocols. It is probably
the key enabling technology for protecting distributed systems, yet it is
surprisingly hard to do right. As we’ve already seen in Chapter 3, ‘Protocols’,
cryptography has often been used to protect the wrong things, or used to
protect them in the wrong way. We’ll see plenty more examples when we start
looking in detail at real applications.
Unfortunately, the computer security and cryptology communities have
drifted apart over the last 25 years. Security people don’t always understand
the available crypto tools, and crypto people don’t always understand the
real-world problems. There are a number of reasons for this, such as different
professional backgrounds (computer science versus mathematics) and different research funding (governments have tried to promote computer security
research while suppressing cryptography). It reminds me of a story told by
a medical friend. While she was young, she worked for a few years in a
country where, for economic reasons, they’d shortened their medical degrees
and concentrated on producing specialists as quickly as possible. One day,
129

130

Chapter 5

■

Cryptography

a patient who’d had both kidneys removed and was awaiting a transplant
needed her dialysis shunt redone. The surgeon sent the patient back from the
theater on the grounds that there was no urinalysis on file. It just didn’t occur
to him that a patient with no kidneys couldn’t produce any urine.
Just as a doctor needs to understand physiology as well as surgery, so
a security engineer needs to be familiar with cryptology as well as computer
security (and much else). This chapter is aimed at people without any training
in cryptology; cryptologists will find little in it that they don’t already know.
As I only have a few dozen pages, and a proper exposition of modern cryptography would run into thousands, I won’t go into much of the mathematics
(there are lots of books that do that; see the end of the chapter for further
reading). I’ll just explain the basic intuitions and constructions that seem
to cause the most confusion. If you have to use cryptography in anything
resembling a novel way, then I strongly recommend that you read a lot more
about it — and talk to some real experts. The security engineer Paul Kocher
remarked, at a keynote speech at Crypto 2007, that you could expect to break
any crypto product designed by ‘any company that doesn’t employ someone
in this room’. There is a fair bit of truth in that.
Computer security people often ask for non-mathematical definitions of
cryptographic terms. The basic terminology is that cryptography refers to
the science and art of designing ciphers; cryptanalysis to the science and
art of breaking them; while cryptology, often shortened to just crypto, is
the study of both. The input to an encryption process is commonly called the
plaintext, and the output the ciphertext. Thereafter, things get somewhat more
complicated. There are a number of cryptographic primitives — basic building
blocks, such as block ciphers, stream ciphers, and hash functions. Block ciphers
may either have one key for both encryption and decryption, in which case
they’re called shared-key (also secret-key or symmetric), or have separate keys
for encryption and decryption, in which case they’re called public-key or
asymmetric. A digital signature scheme is a special type of asymmetric crypto
primitive.
In the rest of this chapter, I will first give some simple historical examples to
illustrate the basic concepts. I’ll then try to fine-tune definitions by introducing
the random oracle model, which many cryptologists use. Finally, I’ll show how
some of the more important cryptographic algorithms actually work, and
how they can be used to protect data.

5.2

Historical Background

Suetonius tells us that Julius Caesar enciphered his dispatches by writing
‘D’ for ‘A’, ‘E’ for ‘B’ and so on [1232]. When Augustus Caesar ascended the
throne, he changed the imperial cipher system so that ‘C’ was now written for

5.2 Historical Background

‘A’, ‘D’ for ‘B’ etcetera. In modern terminology, we would say that he changed
the key from ‘D’ to ‘C’. Remarkably, a similar code was used by Bernardo
Provenzano, allegedly the capo di tutti capi of the Sicilian mafia, who wrote ‘4’
for ‘a’, ‘5’ for ‘b’ and so on. This led directly to his capture by the Italian police
in 2006 after they intercepted and deciphered some of his messages [1034].
The Arabs generalised this idea to the monoalphabetic substitution, in which
a keyword is used to permute the cipher alphabet. We will write the plaintext
in lower case letters, and the ciphertext in upper case, as shown in Figure 5.1:
abcdefghijklmnopqrstuvwxyz
SECURITYABDFGHJKLMNOPQVWXZ
Figure 5.1: Monoalphabetic substitution cipher
OYAN RWSGKFR AN AH RHTFANY MSOYRM OYSH SMSEAC NCMAKO; but breaking

ciphers of this kind is a straightforward pencil and paper puzzle, which you
may have done in primary school. The trick is that some letters, and combinations of letters, are much more common than others; in English the most
common letters are e,t,a,i,o,n,s,h,r,d,l,u in that order. Artificial intelligence
researchers have shown some interest in writing programs to solve monoalphabetic substitutions. Using letter and digram (letter pair) frequencies alone,
they typically succeed with about 600 letters of ciphertext, while smarter
strategies such as guessing probable words can cut this to about 150 letters. A
human cryptanalyst will usually require much less.
There are basically two ways to make a stronger cipher — the stream cipher
and the block cipher. In the former, you make the encryption rule depend on
a plaintext symbol’s position in the stream of plaintext symbols, while in the
latter you encrypt several plaintext symbols at once in a block. Let’s look at
early examples.

5.2.1

An Early Stream Cipher — The Vigenère

This early stream cipher is commonly ascribed to the Frenchman Blaise de
Vigenère, a diplomat who served King Charles IX. It works by adding a key
repeatedly into the plaintext using the convention that ‘A’ = 0, ‘B’ = 1, . . . ,
‘Z’ = 25, and addition is carried out modulo 26 — that is, if the result is greater
than 25, we subtract as many multiples of 26 as are needed to bring is into the
range [0, . . . , 25], that is, [A, . . . , Z]. Mathematicians write this as
C = P + K mod 26
So, for example, when we add P (15) to U (20) we get 35, which we reduce to
9 by subtracting 26.9 corresponds to J, so the encryption of P under the key U
(and of U under the key P) is J. In this notation, Julius Caesar’s system used a

131

132

Chapter 5

■

Cryptography

fixed key K = D1 , while Augustus Caesar’s used K = C and Vigenère used a
repeating key, also known as a running key. Various means were developed
to do this addition quickly, including printed tables and, for field use, cipher
wheels. Whatever the implementation technology, the encryption using a
repeated keyword for the key would look as shown in Figure 5.2:
Plain
Key
Cipher

tobeornottobethatisthequestion
runrunrunrunrunrunrunrunrunrun
KIOVIEEIGKIOVNURNVJNUVKHVMGZIA

Figure 5.2: Vigenère (polyalphabetic substitution cipher)

A number of people appear to have worked out how to solve polyalphabetic
ciphers, from the womaniser Giacomo Casanova to the computing pioneer
Charles Babbage. However the first published solution was in 1863 by Friedrich
Kasiski, a Prussian infantry officer [695]. He noticed that given a long enough
piece of ciphertext, repeated patterns will appear at multiples of the keyword
length.
In Figure 5.2, for example, we see ‘KIOV’ repeated after nine letters, and ‘NU’
after six. Since three divides both six and nine, we might guess a keyword
of three letters. It follows that ciphertext letters one, four, seven and so on
all enciphered under the same keyletter; so we can use frequency analysis
techniques to guess the most likely values of this letter, and then repeat the
process for the second and third letters of the key.

5.2.2

The One-Time Pad

One way to make a stream cipher of this type proof against attacks is for the
key sequence to be as long as the plaintext, and to never repeat. This was proposed by Gilbert Vernam during World War 1 [676]; its effect is that given any
ciphertext, and any plaintext of the same length, there is a key which decrypts
the ciphertext to the plaintext. Regardless of the amount of computation that
opponents can do, they are none the wiser, as all possible plaintexts are just
as likely. This system is known as the one-time pad. Leo Marks’ engaging book
on cryptography in the Special Operations Executive in World War 2 [836]
relates how one-time key material was printed on silk, which agents could
conceal inside their clothing; whenever a key had been used it was torn off
and burnt.
An example should explain all this. Suppose you had intercepted a message
from a wartime German agent which you knew started with ‘Heil Hitler’,
and the first ten letters of ciphertext were DGTYI BWPJA. This means that
1

modulo 23, as the alphabet Caesar used wrote U as V, J as I, and had no W.

5.2 Historical Background

the first ten letters of the one-time pad were wclnb tdefj, as shown in
Figure 5.3:
Plain
Key
Cipher

heilhitler
wclnbtdefj
DGTYIBWPJA

Figure 5.3: A spy’s message

But once he’s burnt the piece of silk with his key material, the spy can claim
that he’s actually a member of the anti-Nazi underground resistance, and the
message actually said ‘Hang Hitler’. This is quite possible, as the key material
could just as easily have been wggsb tdefj, as shown in Figure 5.4:
Cipher
Key
Plain

DGTYIBWPJA
wggsbtdefj
hanghitler

Figure 5.4: What the spy claimed he said

Now we rarely get anything for nothing in cryptology, and the price of the
perfect secrecy of the one-time pad is that it fails completely to protect message
integrity. Suppose for example that you wanted to get this spy into trouble,
you could change the ciphertext to DCYTI BWPJA (Figure 5.5):
Cipher
Key
Plain

DCYTIBWPJA
wclnbtdefj
hanghitler

Figure 5.5: Manipulating the message to entrap the spy

During the Second World War, Claude Shannon proved that a cipher has
perfect secrecy if and only if there are as many possible keys as possible
plaintexts, and every key is equally likely; so the one-time pad is the only kind
of system which offers perfect secrecy [1157, 1158].
The one-time pad is still used for some diplomatic and intelligence traffic,
but it consumes as much key material as there is traffic and this is too expensive
for most applications. It’s more common for stream ciphers to use a suitable
pseudorandom number generator to expand a short key into a long keystream.
The data is then encrypted by exclusive-or’ing the keystream, one bit at a time,
with the data. It’s not enough for the keystream to appear ‘‘random’’ in
the sense of passing the standard statistical randomness tests: it must also
have the property that an opponent who gets his hands on even quite a lot of

133

134

Chapter 5

■

Cryptography

keystream bits should not be able to predict any more of them. I’ll formalise
this more tightly in the next section.
Stream ciphers are commonly used nowadays in hardware applications
where the number of gates has to be minimised to save power. We’ll look
at some actual designs in later chapters, including the A5 algorithm used
to encipher GSM mobile phone traffic (in the chapter on ‘Telecom System
Security’), and the shift register systems used in pay-per-view TV and DVD
CSS (in the chapter on ‘Copyright and Privacy Protection’). However, block
ciphers are more suited for many applications where encryption is done in
software, so let’s look at them next.

5.2.3

An Early Block Cipher — Playfair

One of the best-known early block ciphers is the Playfair system. It was
invented in 1854 by Sir Charles Wheatstone, a telegraph pioneer who
also invented the concertina and the Wheatstone bridge. The reason it’s
not called the Wheatstone cipher is that he demonstrated it to Baron Playfair,
a politician; Playfair in turn demonstrated it to Prince Albert and to Viscount
Palmerston (later Prime Minister), on a napkin after dinner.
This cipher uses a 5 by 5 grid, in which we place the alphabet, permuted by
the key word, and omitting the letter ‘J’ (see Figure 5.6):
P
R
B
H
V

A
S
C
I
W

L
T
D
K
X

M
O
F
Q
Y

E
N
G
U
Z

Figure 5.6: The Playfair enciphering tableau

The plaintext is first conditioned by replacing ‘J’ with ‘I’ wherever it occurs,
then dividing it into letter pairs, preventing double letters occurring in a pair by
separating them with an ‘x’, and finally adding a ‘z’ if necessary to complete the
last letter pair. The example Playfair wrote on his napkin was ‘Lord Granville’s
letter’ which becomes ‘lo rd gr an vi lx le sl et te rz’.
It is then enciphered two letters at a time using the following rules:
if the two letters are in the same row or column, they are replaced by the
succeeding letters. For example, ‘am’ enciphers to ‘LE’
otherwise the two letters stand at two of the corners of a rectangle in
the table, and we replace them with the letters at the other two corners of
this rectangle. For example, ‘lo’ enciphers to ‘MT’.

5.2 Historical Background

We can now encipher our specimen text as follows:
Plain
Cipher

lo rd gr an vi lx le sl et te rz
MT TB BN ES WH TL MP TA LN NL NV

Figure 5.7: Example of Playfair enciphering

Variants of this cipher were used by the British army as a field cipher in
World War 1, and by the Americans and Germans in World War 2. It’s a
substantial improvement on Vigenère as the statistics which an analyst can
collect are of digraphs (letter pairs) rather than single letters, so the distribution
is much flatter and more ciphertext is needed for an attack.
Again, it’s not enough for the output of a block cipher to just look intuitively
‘random’. Playfair ciphertexts look random; but they have the property that if
you change a single letter of a plaintext pair, then often only a single letter of
the ciphertext will change. Thus using the key in Figure 5.7, rd enciphers to
TB while rf enciphers to OB and rg enciphers to NB. One consequence is that
given enough ciphertext, or a few probable words, the table (or an equivalent
one) can be reconstructed [512]. So we will want the effects of small changes
in a block cipher’s input to diffuse completely through its output: changing
one input bit should, on average, cause half of the output bits to change. We’ll
tighten these ideas up in the next section.
The security of a block cipher can be greatly improved by choosing a longer
block length than two characters. For example, the Data Encryption Standard
(DES), which is widely used in banking, has a block length of 64 bits, which
equates to eight ascii characters and the Advanced Encryption Standard (AES),
which is replacing it in many applications, has a block length of twice this. I
discuss the internal details of DES and AES below; for the time being, I’ll just
remark that an eight byte or sixteen byte block size is not enough of itself.
For example, if a bank account number always appears at the same place
in a transaction, then it’s likely to produce the same ciphertext every time a
transaction involving it is encrypted with the same key.
This might allow an opponent to cut and paste parts of two different ciphertexts in order to produce a seemingly genuine but unauthorized transaction.
Suppose a bad man worked for a bank’s phone company, and could intercept
their traffic. If he monitored an enciphered transaction that he knew said ‘‘Pay
IBM $10,000,000’’ he might wire $1,000 to his brother causing the bank computer to insert another transaction saying ‘‘Pay John Smith $1,000’’, intercept
this instruction, and make up a false instruction from the two ciphertexts that
decrypted as ‘‘Pay John Smith $10,000,000’’. So unless the cipher block is as
large as the message, the ciphertext will contain more than one block and we
will usually need some way of binding the blocks together.

135

136

Chapter 5

5.2.4

■

Cryptography

One-Way Functions

The third classical type of cipher is the one-way function. This evolved to
protect the integrity and authenticity of messages, which as we’ve seen
is not protected at all by many simple ciphers where it is often easy to
manipulate the ciphertext in such a way as to cause a predictable change in the
plaintext.
After the invention of the telegraph in the mid-19th century, banks rapidly
became its main users and developed systems for transferring money electronically. Of course, it isn’t the money itself which is ‘wired’ but a payment
instruction, such as:
‘To Lombard Bank, London. Please pay from our account with you no. 1234567890
the sum of £1000 to John Smith of 456 Chesterton Road, who has an account with
HSBC Bank Cambridge no. 301234 4567890123, and notify him that this was
for ‘‘wedding present from Doreen Smith’’. From First Cowboy Bank of Santa
Barbara, CA, USA. Charges to be paid by us.’
Since telegraph messages were relayed from one office to another by human
operators, it was possible for an operator to manipulate a payment message.
In the nineteenth century, banks, telegraph companies and shipping companies developed code books that could not only protect transactions but also
shorten them — which was very important given the costs of international
telegrams at the time. A code book was essentially a block cipher which
mapped words or phrases to fixed-length groups of letters or numbers. So
‘Please pay from our account with you no.’ might become ‘AFVCT’. A competing technology from the 1920s was rotor machines, mechanical cipher devices
which produce a very long sequence of pseudorandom numbers and combine
them with plaintext to get ciphertext; these were independently invented by
a number of people, many of whom dreamed of making a fortune selling
them to the banking industry. Banks weren’t in general interested, but rotor
machines became the main high-level ciphers used by the combatants in
World War 2.
The banks realised that neither mechanical stream ciphers nor code books
protect message authenticity. If, for example, the codeword for ‘1000’ is
‘mauve’ and for ‘1,000,000’ is ‘magenta’, then the crooked telegraph clerk who
can compare the coded traffic with known transactions should be able to figure
this out and substitute one for the other.
The critical innovation, for the banks’ purposes, was to use a code book
but to make the coding one-way by adding the code groups together into
a number called a test key. (Modern cryptographers would describe it as a
hash value or message authentication code, terms I’ll define more carefully
later.)

5.2 Historical Background

Here is a simple example. Suppose the bank has a code book with a table of
numbers corresponding to payment amounts as in Figure 5.8:

x 1000
x 10,000
x 100,000
x 1,000,000

0
14
73
95
53

1
22
38
70
77

2
40
15
09
66

3
87
46
54
29

4
69
91
82
40

5
93
82
63
12

6
71
00
21
31

7
35
29
47
05

8
06
64
36
87

9
58
57
18
94

Figure 5.8: A simple test key system

Now in order to authenticate a transaction for £376,514 we add together 53
(no millions), 54 (300,000), 29 (70,000) and 71 (6,000). (It’s common to ignore
the less significant digits of the amount.) This gives us a test key of 207.
Most real systems were more complex than this; they usually had tables
for currency codes, dates and even recipient account numbers. In the better
systems, the code groups were four digits long rather than two, and in order
to make it harder for an attacker to reconstruct the tables, the test keys were
compressed: a key of ‘7549’ might become ‘23’ by adding the first and second
digits, and the third and fourth digits, and ignoring the carry.
This made such test key systems into one-way functions in that although given
knowledge of the key it was possible to compute a test from a message, it was
not possible to reverse the process and recover a message from a test — the
test just did not contain enough information. Indeed, one-way functions had
been around since at least the seventeenth century. The scientist Robert Hooke
published in 1678 the sorted anagram ‘ceiiinosssttuu’ and revealed two years
later that it was derived from ‘Ut tensio sic uis’ — ‘the force varies as the
tension’, or what we now call Hooke’s law for a spring. (The goal was to
establish priority for the idea while giving him time to continue developing it.)
Test keys are not strong by the standards of modern cryptography. Given
somewhere between a few dozen and a few hundred tested messages, depending on the design details, a patient analyst could reconstruct enough of the
tables to forge a transaction. With a few carefully chosen messages inserted
into the banking system by an accomplice, it’s even easier still. But the banks
got away with it: test keys worked fine from the late nineteenth century
through the 1980’s. In several years working as a bank security consultant, and
listening to elderly bank auditors’ tales over lunch, I only ever heard of two
cases of fraud that exploited it: one external attempt involving cryptanalysis,
which failed because the attacker didn’t understand bank procedures, and one
successful but small fraud involving a crooked staff member. I’ll discuss the
systems which replaced test keys, and the whole issue of how to tie cryptographic authentication mechanisms to procedural protection such as dual

137

138

Chapter 5

■

Cryptography

control, in the chapter on ‘Banking and Bookkeeping’. For the meantime, test
keys are the classic example of a one-way function used for authentication.
Later examples included functions for applications discussed in the previous
chapters, such as storing passwords in a one-way encrypted password file,
and computing a response from a challenge in an authentication protocol.

5.2.5

Asymmetric Primitives

Finally, some modern cryptosystems are asymmetric, in that different keys
are used for encryption and decryption. So, for example, many people publish
on their web page a public key with which people can encrypt messages to
send to them; the owner of the web page can then decrypt them using the
corresponding private key.
There are some pre-computer examples of this too; perhaps the best is the
postal service. You can send me a private message just as simply by addressing
it to me and dropping it into a post box. Once that’s done, I’m the only person
who’ll be able to read it. There are of course many things that can go wrong.
You might get the wrong address for me (whether by error or as a result of
deception); the police might get a warrant to open my mail; the letter might be
stolen by a dishonest postman; a fraudster might redirect my mail without my
knowledge; or a thief might steal the letter from my mailbox. There are similar
things that can go wrong with public key cryptography. False public keys can
be inserted into the system, computers can be hacked, people can be coerced
and so on. We’ll look at these problems in more detail in later chapters.
Another asymmetric application of cryptography is the digital signature. The
idea here is that I can sign a message using a private signature key and then
anybody can check this using my public signature verification key. Again, there
are pre-computer analogues in the form of manuscript signatures and seals;
and again, there is a remarkably similar litany of things that can go wrong,
both with the old way of doing things and with the new.

5.3

The Random Oracle Model

Before delving into the detailed design of modern ciphers, I want to take a few
pages to refine the definitions of the various types of cipher. (Readers who
are phobic about theoretical computer science should skip this section at a
first pass; I’ve included it because a basic grasp of the terminology of random
oracles is needed to decipher many recent research papers on cryptography.)
The random oracle model seeks to formalize the idea that a cipher is ‘good’ if,
when viewed in a suitable way, it is indistinguishable from a random function
of a certain type. I will call a cryptographic primitive pseudorandom if it passes
all the statistical and other tests which a random function of the appropriate

5.3 The Random Oracle Model

type would pass, in whatever model of computation we are using. Of course,
the cryptographic primitive will actually be an algorithm, implemented as an
array of gates in hardware or a program in software; but the outputs should
‘look random’ in that they’re indistinguishable from a suitable random oracle
given the type and the number of tests that our computation model permits.
In this way, we can hope to separate the problem of designing ciphers from
the problem of using them correctly. Mathematicians who design ciphers can
provide evidence that their cipher is pseudorandom. Quite separately, a
computer scientist who has designed a cryptographic protocol can try to
prove that it is secure on the assumption that the crypto primitives used
to implement it are pseudorandom. The process isn’t infallible, as we saw
with proofs of protocol correctness. Theorems can have bugs, just like programs; the problem could be idealized wrongly; and the mathematicians
might be using a different model of computation from the computer scientists. In fact, there is a live debate among crypto researchers about whether
formal models and proofs are valuable [724]. But crypto theory can help us
sharpen our understanding of how ciphers behave and how they can safely
be used.
We can visualize a random oracle as an elf sitting in a black box with a
source of physical randomness and some means of storage (see Figure 5.9) —
represented in our picture by the dice and the scroll. The elf will accept inputs
of a certain type, then look in the scroll to see whether this query has ever
been answered before. If so, it will give the answer it finds there; if not,
it will generate an answer at random by throwing the dice. We’ll further
assume that there is some kind of bandwidth limitation — that the elf will
only answer so many queries every second. This ideal will turn out to be
useful as a way of refining our notions of a stream cipher, a hash function,
a block cipher, a public key encryption algorithm and a digital signature
scheme.

Figure 5.9: The random oracle

139

140

Chapter 5

■

Cryptography

Finally, we can get a useful simplification of our conceptual model by
noting that encryption can be used to protect data across time as well as across
distance. A good example is when we encrypt data before storing it with a
third-party backup service, and may decrypt it later if we have to recover from
a disk crash. In this case, we only need a single encryption/decryption device,
rather than having one at each end of a communications link. For simplicity,
let us assume it is this sort of application we are modelling here. The user takes
a diskette to the cipher machine, types in a key, issues an instruction, and the
data get transformed in the appropriate way. A year later, she comes back to
get the data decrypted and verified.
We shall now look at this model in more detail for various different
cryptographic primitives.

5.3.1

Random Functions — Hash Functions

The first type of random oracle is the random function. A random function
accepts an input string of any length and outputs a random string of fixed
length, say n bits long. So the elf just has a simple list of inputs and outputs,
which grows steadily as it works. (We’ll ignore any effects of the size of the
scroll and assume that all queries are answered in constant time.)
Random functions are our model for one-way functions, also known as
cryptographic hash functions, which have many practical uses. They were first
used in computer systems for one-way encryption of passwords in the 1960s
and, as I mentioned in the chapter on security protocols, are used today in
a number of authentication systems. They are used to compute checksums
on files in forensic applications: presented with a computer seized from a
suspect, you can compute hash values of the files to identify which files are
already known (such as system files) and which are novel (such as user data).
Hash values are also used as a means of checking the integrity of files, as
they will change if a file is corrupted. In messaging applications, hashes are
often known as message digests; given a message M we can pass it through a
pseudorandom function to get a digest, say h(M), which can stand in for the
message in various applications. One example is digital signature: signature
algorithms tend to be slow if the message is long, so it’s usually convenient to
sign a message digest rather than the message itself.
Another application is timestamping. If we want evidence that we possessed
a given electronic document by a certain date, we might submit it to an online
time-stamping service. However, if the document is still secret — for example
an invention which we plan to patent, and for which we merely want to
establish a priority date — then we might not send the timestamping service
the whole document, but just the message digest.

5.3 The Random Oracle Model

5.3.1.1

Properties

The first main property of a random function is one-wayness. Given knowledge
of an input x we can easily compute the hash value h(x), but it is very difficult
given the hash value h(x) to find a corresponding preimage x if one is not
already known. (The elf will only pick outputs for given inputs, not the other
way round.) As the output is random, the best an attacker who wants to invert
a random function can do is to keep on feeding in more inputs until he gets
lucky. A pseudorandom function will have the same properties, or they could
be used to distinguish it from a random function, contrary to our definition. It
follows that a pseudorandom function will also be a one-way function, provided
there are enough possible outputs that the opponent can’t find a desired target
output by chance. This means choosing the output to be an n-bit number
where the opponent can’t do anything near 2n computations.
A second property of pseudorandom functions is that the output will not
give any information at all about even part of the input. Thus a one-way
encryption of the value x can be accomplished by concatenating it with a
secret key k and computing h(x, k). If the hash function isn’t random enough,
though, using it for one-way encryption in this manner is asking for trouble.
A topical example comes from the authentication in GSM mobile phones,
where a 16 byte challenge from the base station is concatenated with a 16 byte
secret key known to the phone into a 32 byte number, and passed through
a hash function to give an 11 byte output [226]. The idea is that the phone
company also knows k and can check this computation, while someone who
eavesdrops on the radio link can only get a number of values of the random
challenge x and corresponding output from h(x, k). So the eavesdropper must
not be able to get any information about k, or compute h(y, k) for a new
input y. But the one-way function used by many phone companies isn’t oneway enough, with the result that an eavesdropper who can pretend to be
a base station and send a phone about 150,000 suitable challenges and get
the responses can compute the key. I’ll discuss this failure in more detail in
section 20.3.2.
A third property of pseudorandom functions with sufficiently long outputs
is that it is hard to find collisions, that is, different messages M1 = M2 with
h(M1 ) = h(M2 ). Unless the opponent can find a shortcut attack (which would
mean the function wasn’t really pseudorandom) then the best way of finding a
collision is to collect a large set of messages Mi and their corresponding hashes
h(Mi ), sort the hashes, and look for a match. If the hash function output is an
n-bit number, so that there are 2n possible hash values, then the number of
hashes the enemy will need to compute before he can expect to find a match
will be about the square root of this, namely 2n/2 hashes. This fact is of huge
importance in security engineering, so let’s look at it more closely.

141

142

Chapter 5

5.3.1.2

■

Cryptography

The Birthday Theorem

The birthday theorem gets its name from the following problem. A maths
teacher asks a typical class of 30 pupils what they think is the probability that
two of them have the same birthday. Most pupils will intuitively think it’s
unlikely, and the maths teacher then asks the pupils to state their birthdays
one after another. As the result seems unlikely to most people, it’s also known
as the ‘birthday paradox’. The odds of a match exceed 50% once 23 pupils have
been called.
The birthday theorem was first invented in the 1930’s to count fish, which
led to its also being known as capture-recapture statistics [1123]. Suppose there
are N fish in a lake and you catch m of them, ring them and throw them back,
then when you first catch a fish you’ve ringed already, m should be ‘about’
the
√ square root of N. The intuitive reason why this holds is that once you have
N samples, each could potentially
√
√ match any of the others, so the2 number of
possible matches is about N x N or N, which is what you need .
This theorem has many applications for the security engineer. For example,
if we have a biometric system which can authenticate a person’s claim to
identity with a probability of only one in a million that two randomly selected
subjects will be falsely identified as the same person, this doesn’t mean that
we can use it as a reliable means of identification in a university with a user
population of twenty thousand staff and students. This is because there will
be almost two hundred million possible pairs. In fact, you expect to find the
first collision — the first pair of people who can be mistaken for each other by
the system — once you have somewhat over a thousand people enrolled.
There are some applications where collision-search attacks aren’t a problem,
such as in challenge-response protocols where an attacker would have to be
able to find the answer to the challenge just issued, and where you can prevent
challenges repeating. (For example, the challenge might be generated by
encrypting a counter.) So in identify-friend-or-foe (IFF) systems, for example,
common equipment has a response length of 48 to 80 bits.
However, there are other applications in which collisions are unacceptable.
In a digital signature application, if it were possible to find collisions with
h(M1 ) = h(M2 ) but M1 = M2 , then a Mafia owned bookstore’s web site might
get you to sign a message M1 saying something like ‘I hereby order a copy
of Rubber Fetish volume 7 for $32.95’ and then present the signature together
with an M2 saying something like ‘I hereby mortgage my house for $75,000
and please make the funds payable to Mafia Holdings Inc., Bermuda’.
For this reason, hash functions used with digital signature schemes generally
have n large enough to make them collision-free, that is, that 2n/2 computations
2 More precisely, the probability that m fish chosen randomly from N fish are different is
β = N(N − 1) . . . (N − m + 1)/Nm which is asymptotically solved by N m2 /2log(1/β) [708].

5.3 The Random Oracle Model

are impractical for an opponent. The two most common are MD5, which has
a 128-bit output and will thus require at most 264 computations to break, and
SHA1 with a 160-bit output and a work factor for the cryptanalyst of at most
280 . However, collision search gives at best an upper bound on the strength
of a hash function, and both these particular functions have turned out to be
disappointing, with cryptanalytic attacks that I’ll describe later in section 5.6.2.
Collisions are easy to find for MD4 and MD5, while for SHA-1 it takes about
260 computations to find a collision — something that a botnet of half a million
machines should be able to do in a few days.
In any case, a pseudorandom function is also often referred to as being
collision free or collision intractable. This doesn’t mean that collisions don’t exist
— they must, as the set of possible inputs is larger than the set of possible outputs — just that you will never find any of them. The (usually
unstated) assumptions are that the output must be long enough, and that the
cryptographic design of the hash function must be sound.

5.3.2 Random Generators — Stream Ciphers
The second basic cryptographic primitive is the random generator, also known
as a keystream generator or stream cipher. This is also a random function, but
unlike in the hash function case it has a short input and a long output. (If we
had a good pseudorandom function whose input and output were a billion
bits long, and we never wanted to handle any objects larger than this, we could
turn it into a hash function by throwing away all but a few hundred bits of the
output, and turn it into a stream cipher by padding all but a few hundred bits
of the input with a constant.) At the conceptual level, however, it’s common to
think of a stream cipher as a random oracle whose input length is fixed while
the output is a very long stream of bits, known as the keystream.
It can be used quite simply to protect the confidentiality of backup data:
we go to the keystream generator, enter a key, get a long file of random bits,
and exclusive-or it with our plaintext data to get ciphertext, which we then
send to our backup contractor. We can think of the elf generating a random
tape of the required length each time he is presented with a new key as input,
giving it to us and keeping a copy of it on his scroll for reference in case he’s
given the same input again. If we need to recover the data, we go back to
the generator, enter the same key, get the same long file of random data, and
exclusive-or it with our ciphertext to get our plaintext data back again. Other
people with access to the keystream generator won’t be able to generate the
same keystream unless they know the key.
I mentioned the one-time pad, and Shannon’s result that a cipher has perfect
secrecy if and only if there are as many possible keys as possible plaintexts, and
every key is equally likely. Such security is called unconditional (or statistical)
security as it doesn’t depend either on the computing power available to the

143

144

Chapter 5

■

Cryptography

opponent, or on there being no future advances in mathematics which provide
a shortcut attack on the cipher.
One-time pad systems are a very close fit for our theoretical model, except
in that they are typically used to secure communications across space rather
than time: there are two communicating parties who have shared a copy of
the randomly-generated keystream in advance. Vernam’s original telegraph
cipher machine used punched paper tape; a modern diplomatic system might
use DVDs, shipped in a tamper-evident container in a diplomatic bag. Various
techniques have been used to do the random generation. Marks describes
how SOE agents’ silken keys were manufactured in Oxford by little old ladies
shuffling counters.
One important problem with keystream generators is that we want to prevent the same keystream being used more than once, whether to encrypt more
than one backup tape or to encrypt more than one message sent on a communications channel. During World War 2, the amount of Russian diplomatic
traffic exceeded the quantity of one-time tape they had distributed in advance
to their embassies, so it was reused. This was a serious blunder. If M1 + K = C1
and M2 + K = C2 , then the opponent can combine the two ciphertexts to get
a combination of two messages: C1 − C2 = M1 − M2 , and if the messages Mi
have enough redundancy then they can be recovered. Text messages do in
fact contain enough redundancy for much to be recovered, and in the case
of the Russian traffic this led to the Venona project in which the US and UK
decrypted large amounts of wartime Russian traffic afterwards and broke up
a number of Russian spy rings. The saying is: ‘Avoid the two-time tape!’
Exactly the same consideration holds for any stream cipher, and the normal
engineering practice when using an algorithmic keystream generator is to
have a seed as well as a key. Each time the cipher is used, we want it to generate
a different keystream, so the key supplied to the cipher should be different.
So if the long-term key which two users share is K, they may concatenate it
with a seed which is a message number N (or some other nonce) and then
pass it through a hash function to form a working key h(K, N). This working
key is the one actually fed to the cipher machine. The nonce may be a separate
pre-agreed key, or it may be generated at random and sent along with the
ciphertext. However, the details of key management can be quite tricky, and
the designer has to watch out for attacks in which a principal is tricked into
synchronising on the wrong key. In effect, a protocol has to be designed to
ensure that both parties can synchronise on the right working key even in the
presence of an adversary.

5.3.3

Random Permutations — Block Ciphers

The third type of primitive, and the most important in modern commercial
cryptography, is the block cipher, which we model as a random permutation.

5.3 The Random Oracle Model

Here, the function is invertible, and the input plaintext and the output
ciphertext are of a fixed size. With Playfair, both input and output are two
characters; with DES, they’re both bit strings of 64 bits. Whatever the number
of symbols and the underlying alphabet, encryption acts on a block of fixed
length. (So if you want to encrypt a shorter input, you have to pad it as with
the final ‘z’ in our Playfair example.)
We can visualize block encryption as follows. As before, we have an elf in a
box with dice and a scroll. This has on the left a column of plaintexts and on
the right a column of ciphertexts. When we ask the elf to encrypt a message,
it checks in the left hand column to see if it has a record of it. If not, it uses
the dice to generate a random ciphertext of the appropriate size (and which
doesn’t appear yet in the right hand column of the scroll), and then writes
down the plaintext/ciphertext pair in the scroll. If it does find a record, it gives
us the corresponding ciphertext from the right hand column.
When asked to decrypt, the elf does the same, but with the function of
the columns reversed: he takes the input ciphertext, checks it (this time on the
right hand scroll) and if he finds it he gives the message with which it was
previously associated. If not, he generates a message at random (which does
not already appear in the left column) and notes it down.
A block cipher is a keyed family of pseudorandom permutations. For each
key, we have a single permutation which is independent of all the others. We
can think of each key as corresponding to a different scroll. The intuitive idea
is that a cipher machine should output the ciphertext given the plaintext and
the key, and output the plaintext given the ciphertext and the key, but given
only the plaintext and the ciphertext it should output nothing.
We will write a block cipher using the notation established for encryption
in the chapter on protocols:
C = {M}K
The random permutation model also allows us to define different types of
attack on block ciphers. In a known plaintext attack, the opponent is just given a
number of randomly chosen inputs and outputs from the oracle corresponding
to a target key. In a chosen plaintext attack, the opponent is allowed to put a
certain number of plaintext queries and get the corresponding ciphertexts. In
a chosen ciphertext attack he gets to make a number of ciphertext queries. In a
chosen plaintext/ciphertext attack he is allowed to make queries of either type.
Finally, in a related key attack he can make queries that will be answered using
keys related to the target key K, such as K + 1 and K + 2.
In each case, the objective of the attacker may be either to deduce the answer
to a query he hasn’t already made (a forgery attack), or to recover the key
(unsurprisingly known as a key recovery attack).
This precision about attacks is important. When someone discovers a vulnerability in a cryptographic primitive, it may or may not be relevant to your

145

146

Chapter 5

■

Cryptography

application. Often it won’t be, but will have been hyped by the media — so
you will need to be able to explain clearly to your boss and your customers
why it’s not a problem. So you have to look carefully to find out exactly what
kind of attack has been found, and what the parameters are. For example,
the first major attack announced on the Data Encryption Standard algorithm
requires 247 chosen plaintexts to recover the key, while the next major attack
improved this to 243 known plaintexts. While these attacks were of great scientific importance, their practical engineering effect was zero, as no practical
systems make that much known (let alone chosen) text available to an attacker.
Such attacks are often referred to as certificational. They can have a commercial
effect, though: the attacks on DES undermined confidence in it and started
moving people to other ciphers. In some other cases, an attack that started off
as certificational has been developed by later ideas into an exploit.
Which sort of attacks you should be worried about depends on your
application. With a broadcast entertainment system, for example, a bad man
can buy a decoder, observe a lot of material and compare it with the enciphered
broadcast signal; so a known-plaintext attack is the main threat. But there are
surprisingly many applications where chosen-plaintext attacks are possible.
Obvious ones include IFF, where the enemy can send challenges of his choice
to any aircraft in range of one of his radars; and ATMs, where if you allow
customers to change their PINs, an attacker can change his PIN through a range
of possible values and observe the enciphered equivalents by wiretapping the
line from the ATM to the bank. A more traditional example is diplomatic
messaging systems, where it’s been known for a host government to give an
ambassador a message to transmit to his capital that’s been specially designed
to help the local cryptanalysts fill out the missing gaps in the ambassador’s
code book [676]. In general, if the opponent can insert any kind of message
into your system, it’s chosen-plaintext attacks you should worry about.
The other attacks are more specialized. Chosen plaintext/ciphertext attacks
may be a worry where the threat is a lunchtime attacker: someone who gets
temporary access to some cryptographic equipment while its authorized
user is out. Related-key attacks are a concern where the block cipher is used
as a building block in the construction of a hash function (which we’ll
discuss below).

5.3.4 Public Key Encryption and Trapdoor One-Way
Permutations
A public-key encryption algorithm is a special kind of block cipher in which the
elf will perform the encryption corresponding to a particular key for anyone
who requests it, but will do the decryption operation only for the key’s owner.
To continue with our analogy, the user might give a secret name to the scroll
that only she and the elf know, use the elf’s public one-way function to

5.3 The Random Oracle Model

compute a hash of this secret name, publish the hash, and instruct the elf to
perform the encryption operation for anybody who quotes this hash.
This means that a principal, say Alice, can publish a key and if Bob wants
to, he can now encrypt a message and send it to her, even if they have never
met. All that is necessary is that they have access to the oracle. There are some
more details that have to be taken care of, such as how Alice’s name can be
bound to the key, and indeed whether it means anything to Bob; I’ll deal with
these later.
A common way of implementing public key encryption is the trapdoor
one-way permutation. This is a computation which anyone can perform, but
which can be reversed only by someone who knows a trapdoor such as a secret
key. This model is like the ‘one-way function’ model of a cryptographic hash
function. Let us state it formally nonetheless: a public key encryption primitive
consists of a function which given a random input R will return two keys, KR
(the public encryption key) and KR−1 (the private decryption key) with the
properties that
1. Given KR, it is infeasible to compute KR−1 (so it’s not possible to compute R either);
2. There is an encryption function {. . .} which, applied to a message M
using the encryption key KR, will produce a ciphertext C = {M}KR ; and
3. There is a decryption function which, applied to a ciphertext C using the
decryption key KR−1 , will produce the original message M = {C}KR−1 .
For practical purposes, we will want the oracle to be replicated at both ends
of the communications channel, and this means either using tamper-resistant
hardware or (more commonly) implementing its functions using mathematics
rather than metal. There are several more demanding models than this, for
example to analyze security in the case where the opponent can get ciphertexts
of his choice decrypted, with the exception of the target ciphertext. But this
will do for now.

5.3.5

Digital Signatures

The final cryptographic primitive which we’ll define here is the digital signature. The basic idea is that a signature on a message can be created by only
one person, but checked by anyone. It can thus perform the sort of function
in the electronic world that ordinary signatures do in the world of paper.
Applications include signing software updates, so that a PC can tell that an
update to Windows was really produced by Microsoft rather than by a villain.
Signature schemes can be deterministic or randomized: in the first, computing
a signature on a message will always give the same result and in the second,
it will give a different result. (The latter is more like handwritten signatures;

147

148

Chapter 5

■

Cryptography

no two are ever alike but the bank has a means of deciding whether a given
specimen is genuine or forged). Also, signature schemes may or may not
support message recovery. If they do, then given the signature, anyone can
recover the message on which it was generated; if they don’t, then the verifier
needs to know or guess the message before he can perform the verification.
(There are further, more specialised, signature schemes such as blind signatures
and threshold signatures but I’ll postpone discussion of them for now.)
Formally, a signature scheme, like public key encryption scheme, has a
keypair generation function which given a random input R will return two
keys, σ R (the private signing key) and VR (the public signature verification
key) with the properties that
1. Given the public signature verification key VR, it is infeasible to compute the private signing key σ R;
2. There is a digital signature function which given a message M and a
private signature key σ R, will produce a signature Sigσ R (M); and
3. There is a signature verification function which, given the signature
Sigσ R (M) and the public signature verification key VR will output TRUE
if the signature was computed correctly with σ R and otherwise output
FALSE.
We can model a simple digital signature algorithm as a random function that
reduces any input message to a one-way hash value of fixed length, followed
by a special kind of block cipher in which the elf will perform the operation
in one direction, known as signature, for only one principal, while in the other
direction, it will perform verification for anybody.
Signature verification can take two forms. In the basic scheme, the elf (or the
signature verification algorithm) only outputs TRUE or FALSE depending on
whether the signature is good. But in a scheme with message recovery, anyone
can input a signature and get back the message corresponding to it. In our
elf model, this means that if the elf has seen the signature before, it will give
the message corresponding to it on the scroll, otherwise it will give a random
value (and record the input and the random output as a signature and message
pair). This is sometimes desirable: when sending short messages over a low
bandwidth channel, it can save space if only the signature has to be sent rather
than the signature plus the message. An example is in the machine-printed
postage stamps, or indicia, being brought into use in many countries: the
stamp may consist of a 2-d barcode with a digital signature made by the postal
meter and which contains information such as the value, the date and the
sender’s and recipient’s post codes. We give some more detail about this at
the end of section 14.3.2.

5.4 Symmetric Crypto Primitives

However, in the general case we do not need message recovery, as the
message to be signed may be of arbitrary length and so we will first pass it
through a hash function and then sign the hash value. As hash functions are
one-way, the resulting compound signature scheme does not have message
recovery — although if the underlying signature scheme does, then the hash
of the message can be recovered from the signature.

5.4

Symmetric Crypto Primitives

Now that we have defined the basic crypto primitives, we will look under the
hood to see how they can be implemented in practice. While most explanations
are geared towards graduate mathematics students, the presentation I’ll give
here is based on one I’ve developed over several years with computer science
students. So I hope it will let the non-mathematician grasp the essentials. In
fact, even at the research level, most of cryptography is as much computer
science as mathematics. Modern attacks on ciphers are put together from
guessing bits, searching for patterns, sorting possible results, and so on rather
than from anything particularly highbrow.
We’ll focus in this section on block ciphers, and then see in the next section
how you can make hash functions and stream ciphers from them, and vice
versa. (In later chapters, we’ll also look at some special-purpose ciphers.)

5.4.1

SP-Networks

Claude Shannon suggested in the 1940’s that strong ciphers could be built
by combining substitution with transposition repeatedly. For example, one
might add some key material to a block of input text, and then shuffle subsets
of the input, and continue in this way a number of times. He described the
properties of a cipher as being confusion and diffusion — adding unknown key
values will confuse an attacker about the value of a plaintext symbol, while
diffusion means spreading the plaintext information through the ciphertext.
Block ciphers need diffusion as well as confusion.
The earliest block ciphers were simple networks which combined substitution and permutation circuits, and so were called SP-networks [681].
Figure 5.10 shows an SP-network with sixteen inputs, which we can imagine
as the bits of a sixteen-bit number, and two layers of four-bit invertible substitution boxes (or S-boxes), each of which can be visualized as a lookup table
containing some permutation of the numbers 0 to 15.
The point of this arrangement is that if we were to implement an arbitrary 16
bit to 16 bit function in digital logic, we would need 220 bits of memory — one

149

150

Chapter 5

■

Cryptography

lookup table of 216 bits for each single output bit. That’s hundreds of thousands
of gates, while a four bit to four bit function takes only 4 x 24 or 64 bits of
memory. One might hope that with suitable choices of parameters, the function
produced by iterating this simple structure would be indistinguishable from a
random 16 bit to 16 bit function to an opponent who didn’t know the value of
the key. The key might consist of some choice of a number of four-bit S-boxes,
or it might be added at each round to provide confusion and the resulting text
fed through the S-boxes to provide diffusion.
Three things need to be done to make such a design secure:
1. the cipher needs to be ‘‘wide’’ enough
2. it needs to have enough rounds, and
3. the S-boxes need to be suitably chosen.

5.4.1.1

Block Size

First, a block cipher which operated on sixteen bit blocks would have rather
limited applicability, as an opponent could just build a dictionary of plaintext
and ciphertext blocks as he observed them. The birthday theorem tells us that
even if the input plaintexts were random, he’d expect to find a match as soon
as he had seen a little over 28 blocks. So a practical block cipher will usually
deal with plaintexts and ciphertexts of 64 bits, 128 bits or even more. So if we
are using four-bit to four-bit S-boxes, we may have 16 of them (for a 64 bit
block size) or 32 of them (for a 128 bit block size).

5.4.1.2

Number of Rounds

Second, we have to have enough rounds. The two rounds in Figure 5.10 are
completely inadequate, as an opponent can deduce the values of the S-boxes
by tweaking input bits in suitable patterns. For example, he could hold the
rightmost 12 bits constant and try tweaking the leftmost four bits, to deduce
the values in the top left S-box. (The attack is slightly more complicated than
this, as sometimes a tweak in an input bit to an S-box won’t produce a change
in any output bit, so we have to change one of its other inputs and tweak
again. But implementing it is still a simple student exercise.)
The number of rounds we require depends on the speed with which data
diffuse through the cipher. In the above simple example, diffusion is very slow
because each output bit from one round of S-boxes is connected to only one
input bit in the next round. Instead of having a simple permutation of the
wires, it is more efficient to have a linear transformation in which each input
bit in one round is the exclusive-or of several output bits in the previous round.
Of course, if the block cipher is to be used for decryption as well as encryption,
this linear transformation will have to be invertible. We’ll see some concrete
examples below in the sections on Serpent and AES.

5.4 Symmetric Crypto Primitives











S-box













S-box









S-box





S-box








S-box












S-box

 



 


 











 



 



 




 
           

S-box








S-box










Figure 5.10: A simple 16-bit SP-network block cipher

5.4.1.3

Choice of S-Boxes

The design of the S-boxes also affects the number of rounds required for
security, and studying bad choices gives us our entry into the deeper theory
of block ciphers. Suppose that the S-box were the permutation that maps the
inputs (0,1,2,. . . ,15) to the outputs (5,7,0,2,4,3,1,6,8,10,15,12,9,11,14,13). Then
the most significant bit of the input would come through unchanged as the
most significant bit of the output. If the same S-box were used in both rounds
in the above cipher, then the most significant bit of the input would pass
through to become the most significant bit of the output. This would usually
be a bad thing; we certainly couldn’t claim that our cipher was pseudorandom.

5.4.1.4

Linear Cryptanalysis

Attacks on real block ciphers are usually harder to spot than in this artificial
example, but they use the same ideas. It might turn out that the S-box had
the property that bit one of the input was equal to bit two plus bit four
of the output; more commonly, there will be linear approximations to an S-box
which hold with a certain probability. Linear cryptanalysis [602, 843] proceeds
by collecting a number of relations such as ‘bit 2 plus bit 5 of the input to the
first S-box is equal to bit 1 plus bit 8 of the output, with probability 13/16’
and then searching for ways to glue them together into an algebraic relation
between input bits, output bits and key bits that holds with a probability
different from one half. If we can find a linear relationship that holds over the
whole cipher with probability p = 0.5 + 1/M, then according to probability
theory we can expect to start recovering keybits once we have about M2
known texts. If the value of M2 for the best linear relationship is greater than
the total possible number of known texts (namely 2n where the inputs and
outputs are n bits wide), then we consider the cipher to be secure against linear
cryptanalysis.

151

152

Chapter 5

5.4.1.5

■

Cryptography

Differential Cryptanalysis

Differential Cryptanalysis [170, 602] is similar but is based on the probability that
a given change in the input to an S-box will give rise to a certain change in the
output. A typical observation on an 8-bit S-box might be that ‘if we flip input
bits 2, 3, and 7 at once, then with probability 11/16 the only output bits that
will flip are 0 and 1’. In fact, with any nonlinear Boolean function, tweaking
some combination of input bits will cause some combination of output bits to
change with a probability different from one half. The analysis procedure is
to look at all possible input difference patterns and look for those values δi ,
δo such that an input change of δi will produce an output change of δo with
particularly high (or low) probability.
As in linear cryptanalysis, we then search for ways to join things up so that
an input difference which we can feed into the cipher will produce a known
output difference with a useful probability over a number of rounds. Given
enough chosen inputs, we will see the expected output and be able to make
deductions about the key. As in linear cryptanalysis, it’s common to consider
the cipher to be secure if the number of texts required for an attack is greater
than the total possible number of different texts for that key. (We have to be
careful though of pathological cases, such as if you had a cipher with a 32-bit
block and a 128-bit key with a differential attack whose success probability
given a single pair was 2−40 . Given a lot of text under a number of keys, we’d
eventually solve for the current key.)
There are a quite a few variants on these two themes. For example, instead of
looking for high probability differences, we can look for differences that can’t
happen (or that happen only rarely). This has the charming name of impossible
cryptanalysis, but it is quite definitely possible against many systems [169].
There are also various specialised attacks on particular ciphers.
Block cipher design involves a number of trade-offs. For example, we can
reduce the per-round information leakage, and thus the required number of
rounds, by designing the rounds carefully. However, a complex design might
be slow in software, or need a lot of gates in hardware, so using simple rounds
but more of them might have been better. Simple rounds may also be easier
to analyze. A prudent designer will also use more rounds than are strictly
necessary to block the attacks known today, in order to give some margin of
safety against improved mathematics in the future. We may be able to show
that a cipher resists all the attacks we know of, but this says little about whether
it will resist the attacks we don’t know of yet. (A general security proof for a
block cipher would appear to imply a proof about an attacker’s computational
powers, which might entail a result such as P = NP that would revolutionize
computer science.)
The point that the security engineer should remember is that block cipher
cryptanalysis is a complex subject about which we have a fairly extensive
theory. Use an off-the-shelf design that has been thoroughly scrutinized

5.4 Symmetric Crypto Primitives

by experts, rather than rolling your own; and if there’s a compelling reason
to use a proprietary cipher (for example, if you want to use a patented design to
stop other people copying a product) then get it reviewed by experts. Cipher
design is not an amateur sport any more.

5.4.1.6

Serpent

As a concrete example, the encryption algorithm ‘Serpent’ is an SP-network
with input and output block sizes of 128 bits. These are processed through 32
rounds, in each of which we first add 128 bits of key material, then pass the text
through 32 S-boxes of 4 bits width, and then perform a linear transformation
that takes each output of one round to the inputs of a number of S-boxes
in the next round. Rather than each input bit in one round coming from a
single output bit in the last, it is the exclusive-or of between two and seven of
them. This means that a change in an input bit propagates rapidly through the
cipher — a so-called avalanche effect which makes both linear and differential
attacks harder. After the final round, a further 128 bits of key material
are added to give the ciphertext. The 33 times 128 bits of key material required
are computed from a user supplied key of up to 256 bits.
This is a real cipher using the structure of Figure 5.10, but modified
to be ‘wide’ enough and to have enough rounds. The S-boxes are chosen to
make linear and differential analysis hard; they have fairly tight bounds on
the maximum linear correlation between input and output bits, and on the
maximum effect of toggling patterns of input bits. Each of the 32 S-boxes in a
given round is the same; this means that bit-slicing techniques can be used to
give a very efficient software implementation on 32-bit processors.
Its simple structure makes Serpent easy to analyze, and it can be shown that
it withstands all the currently known attacks. A full specification of Serpent
is given in [60] and can be downloaded, together with implementations in a
number of languages, from [61].

5.4.2 The Advanced Encryption Standard (AES)
This discussion has prepared us to describe the Advanced Encryption Standard, an algorithm also known as Rijndael after its inventors Vincent Rijmen
and Joan Daemen [342]. This algorithm acts on 128-bit blocks and can use a
key of 128, 192 or 256 bits in length. It is an SP-network; in order to specify it,
we need to fix the S-boxes, the linear transformation between the rounds, and
the way in which the key is added into the computation.
AES uses a single S-box which acts on a byte input to give a byte output.
For implementation purposes it can be regarded simply as a lookup table of
256 bytes; it is actually defined by the equation S(x) = M(1/x) + b over the
field GF(28 ) where M is a suitably chosen matrix and b is a constant. This
construction gives tight differential and linear bounds.

153

154

Chapter 5

■

Cryptography

The linear transformation is based on arranging the 16 bytes of the value
being enciphered in a square and then doing bytewise shuffling and mixing
operations. (AES is descended from an earlier cipher called Square, which
introduced this technique.)
The first step in the linear transformation is the shuffle in which the top row
of four bytes is left unchanged, while the second row is shifted one place to
the left, the third row by two places and the fourth row by three places. The
second step is a column mixing step in which the four bytes in a column are
mixed using a matrix multiplication. This is illustrated in Figure 5.11 which
shows, as an example, how a change in the value of the third byte in the first
column is propagated. The effect of this combination is that a change in the
input to the cipher can potentially affect all of the output after just two rounds.
The key material is added byte by byte after the linear transformation. This
means that 16 bytes of key material are needed per round; they are derived
from the user supplied key material by means of a recurrence relation.
The algorithm uses 10 rounds with 128-bit keys, 12 rounds with 192-bit keys
and 14 rounds with 256-bit keys. These give a reasonable margin of safety; the
best shortcut attacks known at the time of writing (2007) can tackle 7 rounds
for 128-bit keys, and 9 rounds for 192- and 256-bit keys [16]. The general belief
in the block cipher community is that even if advances in the state of the art
do permit attacks on AES with the full number of rounds, they will be purely
certificational attacks in that they will require infeasibly large numbers of texts.
(AES’s margin of safety against attacks that require only feasible numbers of
texts is about 100%.) Although there is no proof of security — whether in the
sense of pseudorandomness, or in the weaker sense of an absence of shortcut
attacks of known types — there is now a high level of confidence that AES is
secure for all practical purposes. The NSA has since 2005 approved AES with
128-bit keys for protecting information up to SECRET and with 256-bit keys
for TOP SECRET.

1

...

1

1

2

2

3

2

3

4

3

4
Shift
row

4
Mix
column

Figure 5.11: The AES linear transformation, illustrated by its effect on byte 3 of the input

5.4 Symmetric Crypto Primitives

Even although I was an author of Serpent which was an unsuccessful finalist
in the AES competition (the winner Rijndael got 86 votes, Serpent 59 votes,
Twofish 31 votes, RC6 23 votes and MARS 13 votes at the last AES conference),
and although Serpent was designed to have an even larger security margin
than Rijndael, I recommend to my clients that they use AES where a generalpurpose block cipher is required. I recommend the 256-bit-key version, and
not because I think that the 10 rounds of the 128-bit-key variant will be broken
anytime soon. Longer keys are better because some key bits often leak in real
products, as I’ll discuss at some length in the chapters on tamper-resistance
and emission security. It does not make sense to implement Serpent as well,
‘just in case AES is broken’: the risk of a fatal error in the algorithm negotiation
protocol is orders of magnitude greater than the risk that anyone will come
up with a production attack on AES. (We’ll see a number of examples later
where using multiple algorithms, or using an algorithm like DES multiple
times, caused something to break horribly.)
The definitive specification of AES is Federal Information Processing Standard 197, and its inventors have written a book describing its design in
detail [342]. Other information, from book errata to links to implementations,
can be found on the AES Lounge web page [16].
One word of warning: the most likely practical attacks on a real implementation of AES include timing analysis and power analysis, both of which
I discuss in Part II in the chapter on emission security. In timing analysis,
the risk is that an opponent observes cache misses and uses them to work
out the key. The latest versions of this attack can extract a key given the
precise measurements of the time taken to do a few hundred cryptographic
operations. In power analysis, an opponent uses measurements of the current
drawn by the device doing the crypto — think of a bank smartcard that a
customer places in a terminal in a Mafia-owned shop. The two overlap; cache
misses cause a device like a smartcard to draw more power — and can also
be observed on remote machines by an opponent who can measure the time
taken to encrypt. The implementation details matter.

5.4.3

Feistel Ciphers

Many block ciphers use a more complex structure, which was invented by
Feistel and his team while they were developing the Mark XII IFF in the late
1950’s and early 1960’s. Feistel then moved to IBM and founded a research
group which produced the Data Encryption Standard, (DES) algorithm, which
is still the mainstay of financial transaction processing security.
A Feistel cipher has the ladder structure shown in Figure 5.12. The input is
split up into two blocks, the left half and the right half. A round function f1 of

155

156

Chapter 5

■

Cryptography

the left half is computed and combined with the right half using exclusive-or
(binary addition without carry), though in some Feistel ciphers addition with
carry is also used. (We use the notation ⊕ for exclusive-or.) Then, a function
f2 of the right half is computed and combined with the left half, and so
on. Finally (if the number of rounds is even) the left half and right half are
swapped.

Right Half

Left Half


•


⊕

f1

⊕

f2

•





...


⊕

f2k

•

























Figure 5.12: The Feistel cipher structure



5.4 Symmetric Crypto Primitives

A notation which you may see for the Feistel cipher is ψ(f , g, h, . . .) where
f , g, h, . . . are the successive round functions. Under this notation, the above
cipher is ψ(f1 , f2 , . . . f2k−1 , f2k ). The basic result that enables us to decrypt a Feistel
cipher — and indeed the whole point of his design — is that:
ψ −1 (f1 , f2 , . . . , f2k−1 , f2k ) = ψ(f2k , f2k−1 , . . . , f2 , f1 )
In other words, to decrypt, we just use the round functions in the reverse
order. Thus the round functions fi do not have to be invertible, and the Feistel
structure lets us turn any one-way function into a block cipher. This means
that we are less constrained in trying to choose a round function with good
diffusion and confusion properties, and which also satisfies any other design
constraints such as code size, table size, software speed, hardware gate count,
and so on.

5.4.3.1

The Luby-Rackoff Result

The key theoretical result on Feistel ciphers was proved by Mike Luby and
Charlie Rackoff in 1988. They showed that if fi were random functions, then
ψ(f1 , f2 , f3 ) was indistinguishable from a random permutation under chosen
plaintext attack, and this result was soon extended to show that ψ(f1 , f2 , f3 , f4 )
was indistinguishable under chosen plaintext/ciphertext attack — in other
words, it was a pseudorandom permutation.
There are a number of technicalities we omit. In engineering terms, the
effect is that given a really good round function, four rounds of Feistel are
enough. So if we have a hash function in which we have confidence, it is
straightforward to construct a block cipher from it: use four rounds of keyed
hash in a Feistel network.

5.4.3.2

DES

The DES algorithm is widely used in banking, government and embedded
applications. For example, it is the standard in automatic teller machine
networks. It is a Feistel cipher, with a 64-bit block and 56-bit key. Its round
function operates on 32-bit half blocks and consists of three operations:
first, the block is expanded from 32 bits to 48;
next, 48 bits of round key are mixed in using exclusive-or;
the result is passed through a row of eight S-boxes, each of which takes a
six-bit input and provides a four-bit output;
finally, the bits of the output are permuted according to a fixed pattern.

157

158

Chapter 5

■

Cryptography

The effect of the expansion, key mixing and S-boxes is shown in Figure 5.13:

Key added
in here
•••

Si – 1

Si

Si + 1

•••

Figure 5.13: The DES round function

The round keys are derived from the user-supplied key by using each user
key bit in twelve different rounds according to a slightly irregular pattern.
A full specification of DES is given in [936]; code can be found in [1125] or
downloaded from many places on the web.
DES was introduced in 1974 and caused some controversy. The most telling
criticism was that the key is too short. Someone who wants to find a 56 bit
key using brute force, that is by trying all possible keys, will have a total
exhaust time of 256 encryptions and an average solution time of half that, namely
255 encryptions. Whit Diffie and Martin Hellman argued in 1977 that a DES
keysearch machine could be built with a million chips, each testing a million
keys a second; as a million is about 220 , this would take on average 215 seconds,
or a bit over 9 hours, to find the key. They argued that such a machine could
be built for $20 million dollars in 1977 [386]. IBM, whose scientists invented
DES, retorted that they would charge the US government $200 million to build
such a machine. (Perhaps both were right.)
During the 1980’s, there were persistent rumors of DES keysearch machines
being built by various intelligence agencies, but the first successful public keysearch attack took place in 1997. In a distributed effort organised
over the net, 14,000 PCs took more than four months to find the key to
a challenge. In 1998, the Electronic Frontier Foundation (EFF) built a DES
keysearch machine called Deep Crack for under $250,000 which broke a
DES challenge in 3 days. It contained 1,536 chips run at 40MHz, each chip
containing 24 search units which each took 16 cycles to do a test decrypt. The
search rate was thus 2.5 million test decryptions per second per search unit, or
60 million keys per second per chip. The design of the cracker is public and can
be found at [423]. By 2006, Sandeep Kumar and colleagues at the universities
of Bochum and Kiel built a machine using 120 FPGAs and costing $10,000,
which could break DES in 7 days on average [755]. A modern botnet with half

5.4 Symmetric Crypto Primitives

a million machines would take a few hours. So the key length of DES is now
definitely inadequate, and banks have for some years been upgrading their
payment systems.
Another criticism of DES was that, since IBM kept its design principles secret
at the request of the US government, perhaps there was a ‘trapdoor’ which
would give them easy access. However, the design principles were published
in 1992 after differential cryptanalysis was invented and published [326].
Their story was that IBM had discovered these techniques in 1972, and the
US National Security Agency (NSA) even earlier. IBM kept the design details
secret at the NSA’s request. We’ll discuss the political aspects of all this
in 24.3.9.1.
We now have a fairly thorough analysis of DES. The best known shortcut
attack, that is, a cryptanalytic attack involving less computation than keysearch,
is a linear attack using 242 known texts. DES would be secure with more than
20 rounds, but for practical purposes its security is limited by its keylength. I
don’t know of any real applications where an attacker might get hold of even
240 known texts. So the known shortcut attacks are not an issue. However, its
growing vulnerability to keysearch makes DES unusable in its original form.
If Moore’s law continues, than by 2020 it might be possible to find a DES key
on a single PC in a few months, so even low-grade systems such as taxi meters
will be vulnerable to brute force-cryptanalysis. As with AES, there are also
attacks based on timing analysis and power analysis, but because of DES’s
structure, the latter are more serious.
The usual way of dealing with the DES keysearch problem is to use
the algorithm multiple times with different keys. Banking networks have
largely moved to triple-DES, a standard since 1999 [936]. Triple-DES does an
encryption, then a decryption, and then a further encryption, all done with
independent keys. Formally:
3DES(k0 , k1 , k2 ; M) = DES(k2 ; DES−1 (k1 ; DES(k0 ; M)))
The reason for this design is that by setting the three keys equal, one gets the
same result as a single DES encryption, thus giving a backwards compatibility
mode with legacy equipment. (Some banking systems use two-key triple-DES
which sets k2 = k0 ; this gives an intermediate step between single and triple
DES). New systems now use AES as of choice, but banking systems are deeply
committed to using block ciphers with an eight-byte block size, because of the
message formats used in the many protocols by which ATMs, point-of-sale
terminals and bank networks talk to each other, and because of the use of block
ciphers to generate and protect customer PINs (which I discuss in Chapter 10).
Triple DES is a perfectly serviceable block cipher for such purposes for the
foreseeable future.
Another way of preventing keysearch (and making power analysis harder) is
whitening. In addition to the 56-bit key, say k0 , we choose two 64-bit whitening

159

160

Chapter 5

■

Cryptography

keys k1 and k2 , xor’ing the first with the plaintext before encryption and the
second with the output of the encryption to get the ciphertext afterwards. This
composite cipher is known as DESX, and is used in the Win2K encrypting file
system. Formally,
DESX(k0 , k1 , k2 ; M) = DES(k0 ; M ⊕ k1 ) ⊕ k2
It can be shown that, on reasonable assumptions, DESX has the properties
you’d expect; it inherits the differential strength of DES but its resistance to
keysearch is increased by the amount of the whitening [717]. Whitened block
ciphers are used in some applications.

5.5

Modes of Operation

In practice, how you use an encryption algorithm is often more important
than which one you pick. An important factor is the ‘mode of operation’, which
specifies how a block cipher with a fixed block size (8 bytes for DES, 16 for
AES) can be extended to process messages of arbitrary length.
There are several standard modes of operation for using a block cipher on
multiple blocks [944]. Understanding them, and choosing the right one for the
job, is an important factor in using a block cipher securely.

5.5.1

Electronic Code Book

In electronic code book (ECB) we just encrypt each succeeding block of
plaintext with our block cipher to get ciphertext, as with the Playfair cipher
I gave above as an example. This is adequate for many simple operations
such as challenge-response and some key management tasks; it’s also used
to encrypt PINs in cash machine systems. However, if we use it to encrypt
redundant data the patterns will show through, letting an opponent deduce
information about the plaintext. For example, if a word processing format has
lots of strings of nulls, then the ciphertext will have a lot of blocks whose value
is the encryption of null characters under the current key.
In one popular corporate email system from the late 1980’s, the encryption
used was DES ECB with the key derived from an eight character password. If
you looked at a ciphertext generated by this system, you saw that a certain block
was far more common than the others — the one corresponding to a plaintext
of nulls. This gave one of the simplest attacks on a fielded DES encryption
system: just encrypt a null block with each password in a dictionary and sort
the answers. You can now break at sight any ciphertext whose password was
one of those in your dictionary.
In addition, using ECB mode to encrypt messages of more than one block
length which have an authenticity requirement — such as bank payment

5.5 Modes of Operation

messages — would be foolish, as messages could be subject to a cut and splice
attack along the block boundaries. For example, if a bank message said ‘Please
pay account number X the sum Y, and their reference number is Z’ then an
attacker might initiate a payment designed so that some of the digits of X
could be replaced with some of the digits of Z.

5.5.2 Cipher Block Chaining
Most commercial applications which encrypt more than one block use cipher
block chaining, or CBC, mode. In it, we exclusive-or the previous block of
ciphertext to the current block of plaintext before encryption (see Figure 5.14).
This mode is effective at disguising any patterns in the plaintext: the
encryption of each block depends on all the previous blocks. The input IV
is an initialization vector, a random number that performs the same function
as a seed in a stream cipher and ensures that stereotyped plaintext message
headers won’t leak information by encrypting to identical ciphertext blocks.
However, an opponent who knows some of the plaintext may be able to
cut and splice a message (or parts of several messages encrypted under the
same key), so the integrity protection is not total. In fact, if an error is inserted
into the ciphertext, it will affect only two blocks of plaintext on decryption,
so if there isn’t any integrity protection on the plaintext, an enemy can insert
two-block garbles of random data at locations of his choice.

5.5.3 Output Feedback
Output feedback (OFB) mode consists of repeatedly encrypting an initial value
and using this as a keystream in a stream cipher of the kind discussed above.
P2

P3


IV ⊕

⊕

⊕







P1

EK

EK

EK

•

•

•


C1


C2


C3

Figure 5.14: Cipher Block Chaining (CBC) mode

...

161

162

Chapter 5

■

Cryptography

Writing IV for the initialization vector or seed, the i-th block of keystream will
be given by
Ki = {. . . {{IV}K }K . . . total of i times}
This is one standard way of turning a block cipher into a stream cipher.
The key K is expanded into a long stream of blocks Ki of keystream. Keystream
is typically combined with the blocks of a message Mi using exclusive-or to
give ciphertext Ci = Mi ⊕ Ki ; this arrangement is sometimes called an additive
stream cipher as exclusive-or is just addition module 2 (and some old hand
systems used addition modulo 26).
All additive stream ciphers have an important vulnerability: they fail to
protect message integrity. I mentioned this in the context of the one-time
pad in section 5.2.2 above, but it’s important to realise that this doesn’t just
affect ‘perfectly secure’ systems but ‘real life’ stream ciphers too. Suppose, for
example, that a stream cipher were used to encipher fund transfer messages.
These messages are very highly structured; you might know, for example, that
bytes 37–42 contained the amount of money being transferred. You could then
carry out the following attack. You cause the data traffic from a local bank to
go via your computer, for example by a wiretap. You go into the bank and
send a modest sum (say $500) to an accomplice. The ciphertext Ci = Mi ⊕ Ki ,
duly arrives in your machine. You know Mi for bytes 37–42, so you know
Ki and can easily construct a modified message which instructs the receiving
bank to pay not $500 but $500,000! This is an example of an attack in depth; it is
the price not just of the perfect secrecy we get from the one-time pad, but of
much more humble stream ciphers too.

5.5.4

Counter Encryption

One possible drawback of feedback modes of block cipher encryption is
latency: feedback modes are hard to parallelize. With CBC, a whole block of
the cipher must be computed between each block input and each block output;
with OFB, we can precompute keystream but storing it requires memory. This
can be inconvenient in very high speed applications, such as protecting traffic
on gigabit backbone links. There, as silicon is cheap, we would rather pipeline
our encryption chip, so that it encrypts a new block (or generates a new block
of keystream) in as few clock ticks as possible.
The simplest solution is often is to generate a keystream by just encrypting
a counter: Ki = {IV + i}K . As before, this is then added to the plaintext to get
ciphertext (so it’s also vulnerable to attacks in depth).
Another problem this mode solves when using a 64-bit block cipher such
as triple-DES on a very high speed link is cycle length. An n-bit block cipher
in OFB mode will typically have a cycle length of 2n/2 blocks, after which the
birthday theorem will see to it that the keystream starts to repeat. (Once we’ve
a little over 232 64-bit values, the odds are that two of them will match.) In

5.5 Modes of Operation

CBC mode, too, the birthday theorem ensures that after about 2n/2 blocks, we
will start to see repeats. Counter mode encryption, however, has a guaranteed
cycle length of 2n rather than 2n/2 .

5.5.5 Cipher Feedback
Cipher feedback, or CFB, mode is another kind of stream cipher. It was
designed to be self-synchronizing, in that even if we get a burst error and drop
a few bits, the system will recover synchronization after one block length. This
is achieved by using our block cipher to encrypt the last n bits of ciphertext,
and then adding one of the output bits to the next plaintext bit.
With decryption, the reverse operation is performed, with ciphertext feeding
in from the right in Figure 5.15. Thus even if we get a burst error and drop a
few bits, as soon as we’ve received enough ciphertext bits to fill up the shift
register, the system will resynchronize.
Cipher feedback is not much used any more. It was designed for use in
military HF radio links which are vulnerable to fading, in the days when
digital electronics were relatively expensive. Now that silicon is cheap, people
use dedicated link layer protocols for synchronization and error correction
rather than trying to combine them with the cryptography.

5.5.6 Message Authentication Code
The next official mode of operation of a block cipher is not used to encipher data,
but to protect its integrity and authenticity. This is the message authentication
code, or MAC. To compute a MAC on a message using a block cipher, we
encrypt it using CBC mode and throw away all the output ciphertext blocks
except the last one; this last block is the MAC. (The intermediate results are
kept secret in order to prevent splicing attacks.)
shift register


EK

plaintext

⊕

•  ciphertext

Figure 5.15: Ciphertext feedback mode (CFB)

163

164

Chapter 5

■

Cryptography

This construction makes the MAC depend on all the plaintext blocks as
well as on the key. It is secure provided the message length is fixed; Mihir
Bellare, Joe Kilian and Philip Rogaway proved that any attack on a MAC
under these circumstances would give an attack on the underlying block
cipher [147].
If the message length is variable, you have to ensure that a MAC computed
on one string can’t be used as the IV for computing a MAC on a different
string, so that an opponent can’t cheat by getting a MAC on the composition of
the two strings. In order to fix this problem, NIST has standardised CMAC, in
which a variant of the key is xor-ed in before the last encryption [945]. (CMAC
is based on a proposal by Tetsu Iwata and Kaoru Kurosawa [649].)
There are other possible constructions of MACs: a common one is to use a
hash function with a key, which we’ll look at in more detail in section 5.6.2.

5.5.7

Composite Modes of Operation

In applications needing both integrity and privacy, the standard procedure
used to be to first calculate a MAC on the message using one key, and then CBC
encrypt it using a different key. (If the same key is used for both encryption
and authentication, then the security of the latter is no longer guaranteed;
cut-and-splice attacks are still possible.)
Recently two further modes of operation have been tackled by NIST that
combine encryption and authentication. The first is CCM, which combines
counter-mode encryption with CBC-MAC authentication. The danger to watch
for here is that the counter values used in encryption must not coincide with the
initialisation vector used in the MAC; the standard requires that the formatting
function prevent this [946].
The second combined mode is Galois Counter Mode (GCM), which has just
been approved at the time of writing (2007). This interesting and innovative
mode is designed to be parallelisable so that it can give high throughput
on fast data links with low cost and low latency. As the implementation is
moderately complex, and the algorithm was approved as this book was in
its final edit, I don’t include the details here, but refer you instead to the
official specification [947]. The telegraphic summary is that the encryption is
performed in a variant of counter mode; the resulting ciphertexts are also
multiplied together with key material and message length information in a
Galois field of 2128 elements to get an authenticator tag. The output is thus
a ciphertext of the same length as the plaintext, plus a tag of typically 128
bits. The tag computation uses a universal hash function which comes from
the theory of unconditionally-secure authentication codes; I’ll describe this in
Chapter 13, ‘Nuclear Command and Control’.

5.6 Hash Functions

Both CCM, and old-fashioned CBC plus CBC MAC, need a completely new
MAC to be computed on the whole message if any bit of it is changed. However, the GCM mode of operation has an interesting incremental property: a
new authenticator and ciphertext can be calculated with an amount of effort
proportional to the number of bits that were changed. GCM is an invention of
David McGrew and John Viega of Cisco; their goal was to create an authenticated encryption mode that is highly parallelisable for use in high-performance
network hardware and that only uses one block cipher operation per block of
plaintext, unlike CCM or the old-fashioned CBC plus CBC-MAC [862]. Now
that GCM has been adopted as a standard, we might expect it to become the
most common mode of operation for the encryption of bulk content.

5.6

Hash Functions

In section 5.4.3.1 I showed how the Luby-Rackoff theorem enables us to
construct a block cipher from a hash function. It’s also possible to construct
a hash function from a block cipher. (In fact, we can also construct hash
functions and block ciphers from stream ciphers — so, subject to some caveats
I’ll discuss in the next section, given any one of these three primitives we can
construct the other two.)
The trick is to feed the message blocks one at a time to the key input of
our block cipher, and use it to update a hash value (which starts off at say
H0 = 0). In order to make this operation non-invertible, we add feedforward:
the (i − 1)st hash value is exclusive or’ed with the output of round i. This is
our final mode of operation of a block cipher (Figure 5.16).
hi−1
•

Mi



E
⊕

hi

Figure 5.16: Feedforward mode (hash function)

165

166

Chapter 5

5.6.1

■

Cryptography

Extra Requirements on the Underlying Cipher

The birthday effect makes another appearance here, in that if a hash function h
is built using an n bit block cipher, it is possible to find two messages M1 = M2
with h(M1 ) = h(M2 ) with about 2n/2 effort (hash slightly more than that many
messages Mi and look for a match). So a 64 bit block cipher is not adequate, as
the cost of forging a message would be of the order of 232 messages, which is
quite practical. A 128-bit cipher such as AES may be just about adequate, and
in fact the AACS content protection mechanism used in the next generation of
DVDs uses ‘AES-H’, the hash function derived from AES in this way.
The birthday limit is not the only way in which the hash function mode of
operation is more demanding on the underlying block cipher than a mode such
as CBC designed for confidentiality. A good illustration comes from a cipher
called Treyfer which was designed to encrypt data using as little memory as
possible in the 8051 microcontrollers commonly found in consumer electronics
and domestic appliances [1371]. (It takes only 30 bytes of ROM.)
Treyfer ‘scavenges’ its S-box by using 256 bytes from the ROM, which may
be code, or even — to make commercial cloning riskier — contain a copyright
message. At each round, it acts on eight bytes of text with eight bytes of key
by adding a byte of text to a byte of key, passing it through the S-box, adding
it to the next byte and then rotating the result by one bit (see Figure 5.17).
This rotation deals with some of the problems that might arise if the S-box has
uneven randomness across its bitplanes (for example, if it contains ascii text

P0

k0

P1

k1

•••

+
S

+
<<<1

Figure 5.17: The basic component of the Treyfer block cipher

5.6 Hash Functions

such as a copyright message). Finally, the algorithm makes up for its simple
round structure and probably less than ideal S-box by having a large number
of rounds (32).
Treyfer can in theory be used for confidentiality (although its effective
keylength is only 44 bits). However, the algorithm does have a weakness that
prevents its use in hash functions. It suffers from a fixed-point attack. Given
any input, there is a fair chance we can find a key which will leave the input
unchanged. We just have to look to see, for each byte of input, whether the
S-box assumes the output which, when added to the byte on the right, has
the effect of rotating it one bit to the right. If such outputs exist for each
of the input bytes, then it’s easy to choose key values which will leave the data
unchanged after one round, and thus after 32. The probability that we can do
this depends on the S-box3 . This means that we can easily find collisions if
Treyfer is used as a hash function. In effect, hash functions have to be based
on block ciphers which withstand chosen-key attacks.

5.6.2 Common Hash Functions and Applications
Algorithms similar to Treyfer have been used in hash functions in key management protocols in some pay-TV systems, but typically they have a modification
to prevent fixed-point attacks, such as a procedure to add in the round number
at each round, or to mix up the bits of the key in some way (a key scheduling
algorithm).
The most commonly used hash functions are all cryptographically suspect.
They are based on variants of a block cipher with a 512 bit key and a block size
of either 128 or 160 bits:
MD4 has three rounds and a 128 bit hash value, and a collision was
found for it in 1998 [394];
MD5 has four rounds and a 128 bit hash value, and a collision was found
for it in 2004 [1315, 1317];
the US Secure Hash Standard has five rounds and a 160 bit hash value,
and it was shown in 2005 that a collision can be found with a computational effort of 269 steps rather than the 280 that one would hope given its
block size [1316].
The block ciphers underlying these hash functions are similar: their round
function is a complicated mixture of the register operations available on 32 bit
processors [1125].
3
Curiously, an S-box which is a permutation is always vulnerable, while a randomly selected
one isn’t quite so bad. In many cipher designs, S-boxes which are permutations are essential or
at least desirable. Treyfer is an exception.

167

168

Chapter 5

■

Cryptography

MD5 was broken by Xiaoyun Wang and her colleagues in 2004 [1315, 1317];
collisions can now be found easily, even between strings containing meaningful
text and adhering to message formats such as those used for digital certificates.
Wang seriously dented SHA the following year, providing an algorithm that
will find collisions in only 269 steps [1316]; and at the Crypto 2007 conference,
the view was that finding a collision should cost about 260 . Volunteers were
being recruited for the task. So it appears that soon a collision will be found
and SHA-1 will be declared ‘broken’.
At the time of writing, the US National Institute of Standards and Technology
(NIST) recommends that people use extended block-size versions of SHA, such
as SHA-256 or SHA-512. The draft FIPS 180-3 allows, though discourages, the
original SHA; it specifies SHA-256 and SHA-512, and also supports 224-bit and
384-bit hashes derived from SHA-256 and SHA-512 respectively by changing
the initial values and truncating the output. The NSA specifies the use of SHA256 or SHA-382 along with AES in its Suite B of cryptographic algorithms for
defense use. NIST is also organising a competition to find a replacement hash
function family [949].
Whether a collision-search algorithm that requires months of work on
hundreds of machines (or a few days on a large botnet) will put any given
application at risk can be a complex question. If bank systems would actually
take a message composed by a customer saying ‘Pay X the sum Y’, hash it and
sign it, then a weak hash function could indeed be exploited: a bad man could
find two messages ‘Pay X the sum Y’ and ‘Pay X the sum Z’ that hashed to
the same value, get one signed, and swap it for the other. But bank systems
don’t work like that. They typically use MACs rather than digital signatures
on actual transactions, relying on signatures only in public-key certificates
that bootstrap key-management protocols; and as the public-key certificates
are generated by trusted CAs using fairly constrained algorithms, there isn’t
an opportunity to insert one text of a colliding pair. Instead you’d have to
find a collision with an externally-given target value, which is a much harder
cryptanalytic task.
Hash functions have many uses. One of them is to compute MACs. A naive
method would be to simply hash the message with a key: MACk (M) = h(k, M).
However the accepted way of doing this, called HMAC, uses an extra step
in which the result of this computation is hashed again. The two hashing
operations are done using variants of the key, derived by exclusive-or’ing
them with two different constants. Thus HMACk (M) = h(k ⊕ A, h(k ⊕ B, M)). A
is constructed by repeating the byte 0x36 as often as necessary, and B similarly
from the byte 0x5C. Given a hash function that may be on the weak side, this
is believed to make exploitable collisions harder to find [741]. HMAC is now
FIPS 198, being replaced by FIPS 198-1.

5.6 Hash Functions

Another use of hash functions is to make commitments that are to be
revealed later. For example, I might wish to timestamp a digital document
in order to establish intellectual priority, but not reveal the contents yet. In
that case, I can submit a hash of the document to a commercial timestamping
service [572]. Later, when I reveal the document, the fact that its hash was
timestamped at a given time establishes that I had written it by then. Again,
an algorithm that generates colliding pairs doesn’t break this, as you have to
have the pair to hand when you do the timestamp. The moral, I suppose, is
that engineers should be clear about whether a given application needs a hash
function that’s strongly collision-resistant.
But even though there may be few applications where the ability to find
collisions could enable a bad guy to steal real money today, the existence of a
potential vulnerability can still undermine a system’s value. In 2005, a motorist
accused of speeding in Sydney, Australia, was acquitted after the New South
Wales Roads and Traffic Authority failed to find an expert to testify that MD5
was secure. The judge was ‘‘not satisfied beyond reasonable doubt that the
photograph [had] not been altered since it was taken’’ and acquitted the
motorist; this ruling was upheld on appeal the following year [964]. So even if
a vulnerability doesn’t present an engineering threat, it can still present a very
real certificational threat.
Finally, before we go on to discuss asymmetric cryptography, there are
two particular uses of hash functions which need mention: key updating and
autokeying.
Key updating means that two or more principals who share a key pass it
through a one-way hash function at agreed times: Ki = h(Ki−1 ). The point is
that if an attacker compromises one of their systems and steals the key, he
only gets the current key and is unable to decrypt back traffic. The chain of
compromise is broken by the hash function’s one-wayness. This property is
also known as backward security.
Autokeying means that two or more principals who share a key hash it
at agreed times with the messages they have exchanged since the last key
change: K+1 i = h(Ki , Mi1 , Mi2 , . . .). The point is that if an attacker compromises
one of their systems and steals the key, then as soon as they exchange a
message which he doesn’t observe or guess, security will be recovered in
that he can no longer decrypt their traffic. Again, the chain of compromise is
broken. This property is known as forward security. It is used, for example, in
EFT payment terminals in Australia [143, 145]. The use of asymmetric crypto
allows a slightly stronger form of forward security, namely that as soon as
a compromised terminal exchanges a message with an uncompromised one
which the opponent doesn’t control, then security can be recovered even if the
message is in plain sight. I’ll describe how this trick works next.

169

170

Chapter 5

5.7

■

Cryptography

Asymmetric Crypto Primitives

The commonly used building blocks in asymmetric cryptography, that is public
key encryption and digital signature, are based on number theory. I’ll give only
a brief overview here, and look in more detail at some of the mechanisms used
in Part II where I discuss applications. (If you find the description assumes too
much mathematics, I’d suggest you skip the following two sections and read
up the material from a cryptography textbook.)
The technique is to make the security of the cipher depend on the difficulty of
solving a certain mathematical problem. The two problems which are used in
almost all fielded systems are factorization (used in most commercial systems)
and discrete logarithm (used in many government systems).

5.7.1

Cryptography Based on Factoring

The prime numbers are the positive whole numbers with no proper divisors; that
is, the only numbers that divide a prime number are 1 and the number itself. By
definition, 1 is not prime; so the primes are {2, 3, 5, 7, 11, . . .}. The fundamental
theorem of arithmetic states that each natural number greater than 1 factors into
prime numbers in a way that is unique up to the order of the factors. It is
easy to find prime numbers and multiply them together to give a composite
number, but much harder to resolve a composite number into its factors. The
largest composite product of two large random primes to have been factorized
to date was RSA-200, a 663-bit number (200 decimal digits), factored in 2005.
This factorization was done on a number of PCs and took the equivalent of 75
years’ work on a single 2.2GHz machine. It is possible for factoring to be done
surreptitiously, perhaps using a botnet; in 2001, when the state of the art was
factoring 512-bit numbers, such a challenge was set in Simon Singh’s ‘Code
Book’ and solved by five Swedish students using several hundred computers
to which they had access [24]. By 2007, 512-bit factorization had entered into
mainstream commerce. From 2003, Intuit had protected its Quicken files with
strong encryption, but left a back door based on a 512-bit RSA key so that they
could offer a key recovery service. Elcomsoft appears to have factored this key
and now offers a competing recovery product.
It is believed that factoring an RSA modulus of 1024 bits would require a
special-purpose machine costing in the range of $10–50m and that would take
a year for each factorization [781]; but I’ve heard of no-one seriously planning
to build such a machine. Many physicists hope that a quantum computer could
be built that would make it easy to factor even large numbers. So, given that
Moore’s law is slowing down and that quantum computers haven’t arrived
yet, we can summarise the state of the art as follows. 1024-bit products of
two random primes are hard to factor and cryptographic systems that rely on

5.7 Asymmetric Crypto Primitives

them are at no immediate risk from low-to-medium budget attackers; NIST
expects them to be secure until 2010, while an extrapolation of the history of
factoring records suggests the first factorization will be published in 2018. So
risk-averse organisations that want keys to remain secure for many years are
already using 2048-bit numbers.
The algorithm commonly used to do public-key encryption and digital
signatures based on factoring is RSA, named after its inventors Ron Rivest,
Adi Shamir and Len Adleman. It uses Fermat’s (little) theorem, which states that
for all primes p not dividing a, ap−1 ≡ 1 (mod p) (proof: take the set {1, 2, . . .,
p − 1} and multiply each of them modulo p by a, then cancel out (p − 1)! each
side). Euler’s function φ(n) is the number of positive integers less than n with
which it has no divisor in common; so if n is the product of two primes pq then
φ(n) = (p − 1)(q − 1) (the proof is similar).
The encryption key is a modulus N which is hard to factor (take N = pq for
two large randomly chosen primes p and q, say of 1024 bits each) plus a public
exponent e that has no common factors with either p − 1 or q − 1. The private
key is the factors p and q, which are kept secret. Where M is the message and
C is the ciphertext, encryption is defined by
C ≡ Me

(mod N)

Decryption is the reverse operation:
√
e
M ≡ C (mod N)
Whoever
knows the private key — the factors p and q of N — can easily cal√
e
culate C (mod N). As φ(N) = (p − 1)(q − 1) and e has no common factors with
φ(N), the key’s owner can find a number d such that de ≡ 1 (mod φ(N)) — she
finds the √value of d separately modulo p − 1 and q − 1, and combines the
e
answers. C (mod N) is now computed as Cd (mod N), and decryption works
because of Fermat’s theorem:
Cd ≡ {Me }d ≡ Med ≡ M1+kφ(N) ≡ M.Mkφ(N) ≡ M.1 ≡ M (mod N)
Similarly, the owner of a private key can operate on a message with this to
produce a signature
Sigd (M) ≡ Md

(mod N)

and this signature can be verified by raising it to the power e mod N (thus,
using e and N as the public signature verification key) and checking that the
message M is recovered:
M ≡ (Sigd (M))e

(mod N)

Neither RSA encryption nor signature is generally safe to use on its own.
The reason is that, as encryption is an algebraic process, it preserves certain
algebraic properties. For example, if we have a relation such as M1 M2 = M3

171

172

Chapter 5

■

Cryptography

that holds among plaintexts, then the same relationship will hold among
ciphertexts C1 C2 = C3 and signatures Sig1 Sig2 = Sig3 . This property is known
as a multiplicative homomorphism; a homomorphism is a function that preserves
some mathematical structure. The homomorphic nature of raw RSA means that
it doesn’t meet the random oracle model definitions of public key encryption
or signature.
Another problem with public-key encryption is that if the plaintexts are
drawn from a small set, such as ‘attack’ or ‘retreat’, and the encryption process
is known to the opponent, then he can precompute possible ciphertexts and
recognise them when they appear. Specific algorithms also have specific
vulnerabilities: with RSA, it’s dangerous to use a small exponent e to encrypt
the same message to multiple recipients, as this can lead to an algebraic
attack. To stop the guessing attack, the low-exponent attack and attacks
based on homomorphism, it’s sensible to add in some randomness, and some
redundancy, into a plaintext block before encrypting it. However, there are
good ways and bad ways of doing this.
In fact, crypto theoreticians have wrestled for decades to analyze all the
things that can go wrong with asymmetric cryptography, and to find ways to
tidy it up. Shafi Goldwasser and Silvio Micali came up with formal models of
probabilistic encryption in which we add randomness to the encryption process,
and semantic security, which means that an attacker cannot get any information
at all about a plaintext M that was encrypted to a ciphertext C, even if he
is allowed to request the decryption of any other ciphertext C not equal
to C [536]. There are a number of constructions that give provable semantic
security, but they tend to be too ungainly for practical use.
The common real-world solution is optimal asymmetric encryption padding
(OAEP), where we concatenate the message M with a random nonce N, and
use a hash function h to combine them:
C1 = M ⊕ h(N)
C2 = N ⊕ h(C1 )
In effect, this is a two-round Feistel cipher that uses h as its round function.
The result, the combination C1 , C2 , is then encrypted with RSA and sent. The
recipient then computes N as C2 ⊕ h(C1 ) and recovers M as C1 ⊕ h(N) [148].
(This construction came with a security proof, in which a mistake was subsequently found [1167, 234], sparking a vigorous debate on the value of
mathematical proofs in security engineering [724].) RSA Data Security, which
for years licensed the RSA algorithm, developed a number of public-key
cryptography standards; PKCS #1 describes OAEP [672].
With signatures, things are slightly simpler. In general, it’s often enough
to just hash the message before applying the private key: Sigd = [h(M)]d
(mod N); PKCS #7 describes simple mechanisms for signing a message

5.7 Asymmetric Crypto Primitives

digest [680]. However, in some applications one might wish to include further
data in the signature block, such as a timestamp, or some randomness in order
to make side-channel attacks harder.
Many of the things that have gone wrong with real implementations have
to do with error handling. Some errors can affect cryptographic mechanisms
directly. The most spectacular example was when Daniel Bleichenbacher found
a way to break the RSA implementation in SSL v 3.0 by sending suitably chosen
ciphertexts to the victim and observing any resulting error messages. If he
can learn from the target whether a given c, when decrypted as cd (mod n),
corresponds to a PKCS #1 message, then he can use this to decrypt or sign
messages [189]. Other attacks have depended on measuring the precise time
taken to decrypt; I’ll discuss these in the chapter on emission security. Yet
others have involved stack overflows, whether by sending the attack code in
as keys, or as padding in poorly-implemented standards. Don’t assume that
the only attacks on your crypto code will be doing cryptanalysis.

5.7.2 Cryptography Based on Discrete Logarithms
While RSA is used in most web browsers in the SSL protocol, and in the SSH
protocol commonly used for remote login to computer systems, there are other
products, and many government systems, which base public key operations on
discrete logarithms. These come in a number of flavors, some using ‘normal’
arithmetic while others use mathematical structures called elliptic curves. I’ll
explain the normal case. The elliptic variants use essentially the same idea but
the implementation is more complex.
A primitive root modulo p is a number whose powers generate all the nonzero
numbers mod p; for example, when working modulo 7 we find that 52 = 25
which reduces to 4 (modulo 7), then we can compute 53 as 52 × 5 or 4 × 5
which is 20, which reduces to 6 (modulo 7), and so on, as in Figure 5.18:
51
52 =
53 ≡
54 ≡
55 ≡
56 ≡

25
4x5
6x5
2x5
3x5

=5
≡4
≡6
≡2
≡3
≡1

(mod 7)
(mod 7)
(mod 7)
(mod 7)
(mod 7)
(mod 7)

Figure 5.18: Example of discrete logarithm calculations

Thus 5 is a primitive root modulo 7. This means that given any y, we can
always solve the equation y = 5x (mod 7); x is then called the discrete logarithm
of y modulo 7. Small examples like this can be solved by inspection, but for a
large random prime number p, we do not know how to do this computation.

173

174

Chapter 5

■

Cryptography

So the mapping f : x → gx (mod p) is a one-way function, with the additional
properties that f (x + y) = f (x)f (y) and f (nx) = f (x)n . In other words, it is a
one-way homomorphism. As such, it can be used to construct digital signature
and public key encryption algorithms.

5.7.2.1

Public Key Encryption — Diffie Hellman and ElGamal

To understand how discrete logarithms can be used to build a public-key
encryption algorithm, bear in mind that we want a cryptosystem which does
not need the users to start off with a shared secret key. Consider the following
‘classical’ scenario.
Imagine that Anthony wants to send a secret to Brutus, and the only
communications channel available is an untrustworthy courier (say, a slave
belonging to Caesar). Anthony can take the message, put it in a box, padlock it,
and get the courier to take it to Brutus. Brutus could then put his own padlock
on it too, and have it taken back to Anthony. He in turn would remove his
padlock, and have it taken back to Brutus, who would now at last open it.
Exactly the same can be done using a suitable encryption function that
commutes, that is, has the property that {{M}KA }KB = {{M}KB }KA . Alice can take
the message M and encrypt it with her key KA to get {M}KA which she sends
to Bob. Bob encrypts it again with his key KB getting {{M}KA }KB . But the
commutativity property means that this is just {{M}KB }KA , so Alice can decrypt
it using her key KA getting {M}KB . She sends this to Bob and he can decrypt
it with KB, finally recovering the message M. The keys KA and KB might be
long-term keys if this mechanism were to be used as a conventional public-key
encryption system, or they might be transient keys if the goal were to establish
a key with forward secrecy.
How can a suitable commutative encryption be implemented? The one-time
pad does commute, but is not suitable here. Suppose Alice chooses a random
key xA and sends Bob M ⊕ xA while Bob returns M ⊕ xB and Alice finally
sends him M ⊕ xA ⊕ xB, then an attacker can simply exclusive-or these three
messages together; as X ⊕ X = 0 for all X, the two values of xA and xB both
cancel our leaving as an answer the plaintext M.
The discrete logarithm problem comes to the rescue. If the discrete log
problem based on a primitive root modulo p is hard, then we can use discrete
exponentiation as our encryption function. For example, Alice encodes her
message as the primitive root g, chooses a random number xA, calculates
gxA modulo p and sends it, together with p, to Bob. Bob likewise chooses
a random number xB and forms gxAxB modulo p, which he passes back to
Alice. Alice can now remove her exponentiation: using Fermat’s theorem, she
calculates gxB = (gxAxB )(p−xA) (mod p) and sends it to Bob. Bob can now remove
his exponentiation, too, and so finally gets hold of g. The security of this scheme
depends on the difficulty of the discrete logarithm problem. In practice, it is

5.7 Asymmetric Crypto Primitives

tricky to encode a message to be a primitive root; but there is a much simpler
means of achieving the same effect. The first public key encryption scheme
to be published, by Whitfield Diffie and Martin Hellman in 1976, has a fixed
primitive root g and uses gxAxB modulo p as the key to a shared-key encryption
system. The values xA and xB can be the private keys of the two parties.
Let’s see how this might provide a public-key encryption system. The prime
p and generator g are common to all users. Alice chooses a secret random
number xA, calculates yA = gxA and publishes it opposite her name in the
company phone book. Bob does the same, choosing a random number xB and
publishing yB = gxB . In order to communicate with Bob, Alice fetches yB from
the phone book, forms yBxA which is just gxAxB , and uses this to encrypt the
message to Bob. On receiving it, Bob looks up Alice’s public key yA and forms
yAxB which is also equal to gxAxB , so he can decrypt her message.
Slightly more work is needed to provide a full solution. Some care is needed
when choosing the parameters p and g; and there are several other details
which depend on whether we want properties such as forward security.
Variants on the Diffie-Hellman theme include the US government key exchange
algorithm (KEA) [939], used in network security products such as the Fortezza
card, and the so-called Royal Holloway protocol, which is used by the UK
government [76].
Of course, one of the big problems with public-key systems is how to be
sure that you’ve got a genuine copy of the phone book, and that the entry
you’re interested in isn’t out of date. I’ll discuss that in section 5.7.5.

5.7.2.2

Key Establishment

Mechanisms for providing forward security in such protocols are of independent interest, As before, let the prime p and generator g be common to all
users. Alice chooses a random number RA , calculates gRA and sends it to Bob;
Bob does the same, choosing a random number RB and sending gRB to Alice;
they then both form gRA RB , which they use as a session key (Figure 5.19).
A→B:
B→A:
A→B:

gRA (mod p)
gRB (mod p)
{M}gRA RB

Figure 5.19: The Diffie-Hellman key exchange protocol

Alice and Bob can now use the session key gRA RB to encrypt a conversation.
They have managed to create a shared secret ‘out of nothing’. Even if an
opponent had obtained full access to both their machines before this protocol
was started, and thus knew all their stored private keys, then provided some
basic conditions were met (e.g., that their random number generators were

175

176

Chapter 5

■

Cryptography

not predictable) the opponent could still not eavesdrop on their traffic. This
is the strong version of the forward security property to which I referred in
section 5.6.2. The opponent can’t work forward from knowledge of previous
keys which he might have obtained. Provided that Alice and Bob both destroy
the shared secret after use, they will also have backward security: an opponent
who gets access to their equipment subsequently cannot work backward to
break their old traffic.
But this protocol has a small problem: although Alice and Bob end up with
a session key, neither of them has any idea who they share it with.
Suppose that in our padlock protocol Caesar had just ordered his slave
to bring the box to him instead, and placed his own padlock on it next to
Anthony’s. The slave takes the box back to Anthony, who removes his padlock,
and brings the box back to Caesar who opens it. Caesar can even run two
instances of the protocol, pretending to Anthony that he’s Brutus and to Brutus
that he’s Anthony. One fix is for Anthony and Brutus to apply their seals to
their locks.
With the vanilla Diffie-Hellman protocol, the same idea leads to a middleperson attack. Charlie intercepts Alice’s message to Bob and replies to it; at
the same time, he initiates a key exchange with Bob, pretending to be Alice.
He ends up with a key gRA RC which he shares with Alice, and another key gRB RC
which he shares with Bob. So long as he continues to sit in the middle of the
network and translate the messages between them, they may have a hard time
detecting that their communications are compromised. The usual solution is
to authenticate transient keys, and there are various possibilities.
In one secure telephone product, the two principals would read out an eight
digit hash of the key they had generated and check that they had the same
value before starting to discuss classified matters. A more general solution is
for Alice and Bob to sign the messages that they send to each other.
A few other details have to be got right, such as a suitable choice of the
values p and g. There’s some non-trivial mathematics behind this, which is best
left to specialists. There are also many things that can go wrong in implementations — examples being software that will generate or accept very weak keys
and thus give only the appearance of protection; programs that leak the key by
the amount of time they take to decrypt; and software vulnerabilities leading
to stack overflows and other nasties. Nonspecialists implementing public-key
cryptography should consult up-to-date standards documents and/or use
properly accredited toolkits.

5.7.2.3

Digital Signature

Suppose that the base p and the generator g are public values chosen in some
suitable way, and that each user who wishes to sign messages has a private
signing key X and a public signature verification key Y = gX . An ElGamal

5.7 Asymmetric Crypto Primitives

signature scheme works as follows. Choose a message key k at random, and
form r = gk (mod p). Now form the signature s using a linear equation in k, r,
the message M and the private key X. There are a number of equations that
will do; the particular one that happens to be used in ElGamal signatures is
rX + sk = M
So s is computed as s = (M − rX)/k; this is done modulo φ(p). When both
sides are passed through our one-way homomorphism f (x) = gx mod p we get:
grX gsk ≡ gM
or
Yr rs ≡ gM
An ElGamal signature on the message M consists of the values r and s, and
the recipient can verify it using the above equation.
A few more details need to be fixed up to get a functional digital signature
scheme. As before, bad choices of p and g can weaken the algorithm. We will
also want to hash the message M using a hash function so that we can sign
messages of arbitrary length, and so that an opponent can’t use the algorithm’s
algebraic structure to forge signatures on messages that were never signed.
Having attended to these details and applied one or two optimisations, we get
the Digital Signature Algorithm (DSA) which is a US standard and widely used
in government applications.
DSA (also known as DSS, for Digital Signature Standard) assumes a prime
p of typically 1024 bits, a prime q of 160 bits dividing (p − 1), an element g of
order q in the integers modulo p, a secret signing key x and a public verification
key y = gx . The signature on a message M, Sigx (M), is (r, s) where
r ≡ (gk

(mod p))

(mod q)

s ≡ (h(M) − xr)/k

(mod q)

The hash function used here is SHA1.
DSA is the classic example of a randomized digital signature scheme without
message recovery. The standard has changed somewhat with faster computers,
as variants of the algorithm used to factor large numbers can also be used to
compute discrete logarithms modulo bases of similar size4 . Initially the prime
p could be in the range 512–1024 bits, but this was changed to 1023–1024
bits in 2001 [941]; the proposed third-generation standard will allow primes
p in the range 1024–3072 bits and q in the range 160–256 bits [942]. Further
tweaks to the standard are also foreseeable after a new hash function standard
is adopted.
4

Discrete log efforts lag slightly behind, with a record set in 2006 of 440 bits.

177

178

Chapter 5

5.7.3

■

Cryptography

Special Purpose Primitives

Researchers have discovered a large number of public-key and signature
primitives with special properties. Two that have so far appeared in real
products are threshold cryptography and blind signatures.
Threshold crypto is a mechanism whereby a signing key, or a decryption key,
can be split up among n principals so that any k out of n can sign a message (or
decrypt). For k = n the construction is easy. With RSA, for example, you can
split up the private key d as d = d1 + d2 + . . . + dn . For k < n it’s slightly more
complex (but not much — you use the Lagrange interpolation formula) [382].
Threshold signatures are used in systems where a number of servers process
transactions independently and vote independently on the outcome; they
could also be used to implement business rules such as ‘a check may be signed
by any two of the seven directors’.
Blind signatures are a way of making a signature on a message without
knowing what the message is. For example, if we are using RSA, I can take a
random number R, form Re M (mod n), and give it to the signer who computes
(Re M)d = R.Md (mod n). When he gives this back to me, I can divide out R
to get the signature Md . Now you might ask why on earth someone would
want to sign a document without knowing its contents, but there are indeed
applications.
The first was in digital cash; a bank might want to be able to issue anonymous
payment tokens to customers, and this has been done by getting it to sign
‘digital coins’ without knowing their serial numbers. In such a system, the
bank might agree to honour for $10 any string M with a unique serial number
and a specified form of redundancy, bearing a signature that verified as correct
using the public key (e, n). The blind signature protocol shows how a customer
can get a bank to sign a coin without the banker knowing its serial number. The
effect is that the digital cash can be anonymous for the spender. (There are a few
technical details that need to be sorted out, such as how you detect people who
spend the same coin twice; but these are fixable.) Blind signatures and digital
cash were invented by Chaum [285], along with much other supporting digital
privacy technology which I’ll discuss later [284]. They were used briefly in pilot
projects for road tolls in the Netherlands and for electronic purses in Brussels,
but failed to take off on a broader scale because of patent issues and because
neither banks nor governments really want payments to be anonymous: the
anti-money-laundering regulations nowadays restrict anonymous payment
services to rather small amounts. Anonymous digital credentials are now
talked about, for example, in the context of ‘identity management’: the TPM
chip on your PC motherboard might prove something about you (such as your
age) without actually revealing your name.
Researchers continue to suggest new applications for specialist public key
mechanisms. A popular candidate is in online elections, which require a

5.7 Asymmetric Crypto Primitives

particular mixture of anonymity and accountability. Voters want to be sure
that their votes have been counted, but it’s also desirable that they should
not be able to prove which way they voted to anybody else; if they can, then
vote-buying and intimidation become easier.

5.7.4

Elliptic Curve Cryptography

Finally, discrete logarithms and their analogues exist in many other mathematical structures; thus for example elliptic curve cryptography uses discrete
logarithms on an elliptic curve — a curve given by an equation like y2 =
x3 + ax + b. These curves have the property that you can define an addition
operation on them and use it for cryptography; the algebra gets a bit complex
and a general book like this isn’t the place to set it out. However, elliptic curve
cryptosystems are interesting for two reasons.
First, they give versions of the familiar primitives such as Diffie-Hellmann
key exchange and the Digital Signature Algorithm that use less computation,
and also have slightly shorter variables; both can be welcome in constrained
environments such as smartcards and embedded processors. Elliptic curve
cryptography is used, for example, in the rights-management mechanisms of
Windows Media Player, and has been adopted as a standard by the NSA for
use in defense systems.
Second, some elliptic curves have a bilinear pairing which Dan Boneh and
Matt Franklin used to construct cryptosystems where your public key is your
name [207]. Recall that in RSA and Diffie-Hellmann, the user chose his private
key and then computed a corresponding public key. In a so-called identitybased cryptosystem, you choose your identity then go to a central authority that
issues you with a private key corresponding to that identity. There is a global
public key, with which anyone can encrypt a message to your identity; you
can decrypt this using your private key. Earlier, Adi Shamir had discovered
identity-based signature schemes that allow you to sign messages using a private
key so that anyone can verify the signature against your name [1147]. In both
cases, your private key is computed by the central authority using a systemwide private key known only to itself. Identity-based primitives could have
interesting implications for specialist systems, but in the context of ordinary
public-key and signature systems they achieve much the same result as the
certification of public keys, which I’ll discuss next.

5.7.5

Certification

Now that we can do public-key encryption and digital signature, we need
some mechanism to bind users to keys. The approach proposed by Diffie and
Hellman when they invented digital signatures was to have a directory of the
public keys of a system’s authorized users, like a phone book. A more common

179

180

Chapter 5

■

Cryptography

solution, due to Loren Kohnfelder, is for a certification authority (CA) to sign the
users’ public encryption and/or signature verification keys giving certificates
that contain the user’s name, attributed such as authorizations, and public
keys. The CA might be run by the local system administrator; or it might be a
third party service such as Verisign whose business is to sign public keys after
doing some due diligence about whether they belong to the principals named
in them.
A certificate might be described symbolically as
CA = SigKS (TS , L, A, KA , VA )

(5.1)

where (using the same notation as with Kerberos) TS is the certificate’s starting date and time, L is the length of time for which it is valid, A is the user’s
name, KA is her public encryption key, and VA is her public signature verification key. In this way, only the administrator’s public signature verification
key needs to be communicated to all principals in a trustworthy manner.
Certification is hard, for a whole lot of reasons. I’ll discuss different aspects
later — naming in Chapter 6, ‘Distributed Systems’, public-key infrastructures
in Chapter 21, ‘Network Attack and Defense’, and the policy aspects in Part III.
Here I’ll merely point out that the protocol design aspects are much harder
than they look.
One of the first proposed public-key protocols was due to Dorothy Denning
and Giovanni Sacco, who in 1982 proposed that two users, say Alice and Bob,
set up a shared key KAB as follows. When Alice first wants to communicate
with Bob, she goes to the certification authority and gets current copies of
public key certificates for herself and Bob. She then makes up a key packet
containing a timestamp TA , a session key KAB and a signature, which she
computes on these items using her private signing key. She then encrypts this
whole bundle under Bob’s public key and ships it off to him. Symbolically,
A → B : CA , CB , {TA , KAB , SigKA (TA , KAB )}KB

(5.2)

In 1994, Martı́n Abadi and Roger Needham pointed out that this protocol
is fatally flawed [2]. Bob, on receiving this message, can masquerade as Alice
for as long as Alice’s timestamp TA remains valid! To see how, suppose that
Bob wants to masquerade as Alice to Charlie. He goes to Sam and gets a
fresh certificate CC for Charlie, and then strips off the outer encryption {. . .}KB
from message 3 in the above protocol. He now re-encrypts the signed key
packet TA , KAB , SigKA (TA , KAB ) with Charlie’s public key — which he gets from
CC — and makes up a bogus message 3:
B → C : CA , CC , {TA , KAB , SigKA (TA , KAB )}KC

(5.3)

It is quite alarming that such a simple protocol — essentially, a one line
program — should have such a serious flaw remain undetected for so long.
With a normal program of only a few lines of code, you might expect to find a

5.7 Asymmetric Crypto Primitives

bug in it by looking at it for a minute or two. In fact, public key protocols are
if anything harder to design than protocols that use shared-key encryption,
as they are prone to subtle and pernicious middleperson attacks. This further
motivates the use of formal methods to prove that protocols are correct.
Often, the participants’ names aren’t the most important things the authentication mechanism has to establish. In the STU-III secure telephone used by
the US government and defense contractors, there is a protocol for establishing
transient keys with forward and backward security; to exclude middleperson
attacks, users have a crypto ignition key, a portable electronic device that they
plug into the phone to identify not just their names, but their security clearance
level. In general, textbooks tend to talk about identification as the main goal
of authentication and key management protocols; but in real life, it’s usually
authorization that matters. This is more complex, as it starts to introduce
assumptions about the application into the protocol design. (In fact, the NSA
security manual emphasises the importance of always knowing whether there
is an uncleared person in the room. The STU-III design is a natural way of
extending this to electronic communications.)
One serious weakness of relying on public-key certificates is the difficulty of
getting users to understand all their implications and manage them properly,
especially where they are not an exact reimplementation of a familiar manual
control system [357]. There are many other things that can go wrong with
certification at the level of systems engineering, which I’ll start to look at in
the next chapter.

5.7.6 The Strength of Asymmetric Cryptographic
Primitives
In order to provide the same level of protection as a symmetric block cipher,
asymmetric cryptographic primitives generally require at least twice the block
length. Elliptic curve systems appear to achieve this bound; a 128-bit elliptic
scheme could be about as hard to break as a 64-bit block cipher with a 64-bit
key; and the only public-key encryption schemes used in the NSA’s Suite B of
military algorithms are 256- and 384-bit elliptic curve systems. The commoner
schemes, based on factoring and discrete log, are less robust because there
are shortcut attack algorithms such as the number field sieve that exploit
the fact that some integers are smooth, that is, they have a large number of
small factors. When I wrote the first edition of this book in 2000, the number
field sieve had been used to attack keys up to 512 bits, a task comparable in
difficulty to keysearch on 56-bit DES keys; by the time I rewrote this chapter
for the second edition in 2007, 64-bit symmetric keys had been brute-forced,
and the 663-bit challenge number RSA-200 had been factored. The advance
in factoring has historically been due about equally to better hardware and
better algorithms. I wrote in 2000 that ‘The current consensus is that private

181

182

Chapter 5

■

Cryptography

keys for RSA and for standard discrete log systems should be at least 1024 bits
long, while 2048 bits gives some useful safety margin’; now in 2007, 1024-bit
RSA is widely believed to give about the same protection as 80-bit symmetric
keys, and designers are starting to move to 2048 bits for keys intended to last
many years. As I mentioned above, an extrapolation of recent factoring results
suggests that it might be a decade before we see a 1024-bit challenge factored —
although with Moore’s law starting to slow down, it might take much longer.
No-one really knows. (However I expect to see 768-bit RSA factored within a
few years.)
There has been much research into quantum computers — devices that perform a large number of computations simultaneously using superposed
quantum states. Peter Shor has shown that if a sufficiently large quantum
computer can be built, then both factoring and discrete logarithm computations will become easy [1165]. So far only very small quantum computers
can be built; factoring 15 is about the state of the art in 2007. Many people
are sceptical about whether the technology can be scaled up to threaten real
systems. But if it does, then asymmetric cryptography may have to change
radically. So it is fortunate that many of the things we currently do with asymmetric mechanisms can also be done with symmetric ones; most authentication
protocols in use could be redesigned to use variants on Kerberos.

5.8

Summary

Many ciphers fail because they’re used improperly, so we need a clear model
of what a cipher does. The random oracle model provides a useful intuition:
we assume that each new value returned by the encryption engine is random
in the sense of being statistically independent of all the different outputs seen
before.
Block ciphers for symmetric key applications can be constructed by the
careful combination of substitutions and permutations; for asymmetric applications such as public key encryption and digital signature one uses number
theory. In both cases, there is quite a large body of mathematics. Other kinds
of ciphers — stream ciphers and hash functions — can be constructed from
block ciphers by using them in suitable modes of operation. These have
different error propagation, pattern concealment and integrity protection
properties.
The basic properties that the security engineer needs to understand are not
too difficult to grasp, though there are many subtle things that can go wrong.
In particular, it is surprisingly hard to build systems that are robust even
when components fail (or are encouraged to) and where the cryptographic
mechanisms are well integrated with other measures such as access control
and physical security. I’ll return to this repeatedly in later chapters.

Further Reading

Research Problems
There are many active threads in cryptography research. Many of them are
where crypto meets a particular branch of mathematics (number theory,
algebraic geometry, complexity theory, combinatorics, graph theory, and
information theory). The empirical end of the business is concerned with
designing primitives for encryption, signature and composite operations,
and which perform reasonably well on available platforms. The two meet
in the study of subjects ranging from cryptanalysis, through the search for
primitives that combine provable security properties with decent performance,
to attacks on public key protocols. Research is more driven by the existing body
of knowledge than by applications, though there are exceptions: copyright
protection concerns and ‘Trusted Computing’ have been a stimulus in recent
years, as was the US government’s competition in the late 1990s to find an
Advanced Encryption Standard.
The best way to get a flavor of what’s going on is to read the last few years’
proceedings of research conferences such as Crypto, Eurocrypt, Asiacrypt,
CHES and Fast Software Encryption — all published by Springer in their
Lecture Notes on Computer Science series.

Further Reading
The classic papers by Whit Diffie and Martin Hellman [385] and by Ron
Rivest, Adi Shamir and Len Adleman [1078] are the closest to required reading
in this subject. The most popular introduction is Bruce Schneier’s Applied
Cryptography [1125] which covers a lot of ground at a level a non-mathematician
can understand, but is slightly dated. Alfred Menezes, Paul van Oorshot and
Scott Vanstone’s Handbook of Applied Cryptography [872] is the closest to a
standard reference book on the mathematical detail. For an appreciation of the
recent history of cryptanalysis, try Mark Stamp and Richard Low’s ‘Applied
Cryptanalysis’ [1214]: this has recent attacks on fielded ciphers such as PKZIP,
RC4, CMEA and MD5.
There are many more specialised references. The bible on differential cryptanalysis is a book by its inventors Eli Biham and Adi Shamir [170], while a good
short tutorial on linear and differential cryptanalysis was written by Howard
Heys [602]. A textbook by Doug Stinson has another detailed explanation of
linear cryptanalysis [1226]; and the modern theory of block ciphers can be
traced through the papers in the Fast Software Encryption conference series. The
original book on modes of operation is by Carl Meyer and Steve Matyas [880].
Neal Koblitz has a good basic introduction to the mathematics behind public
key cryptography [723]; and the number field sieve is described in [780].

183

184

Chapter 5

■

Cryptography

There’s a shortage of good books on the random oracle model and on
theoretical cryptology in general: all the published texts I’ve seen are very
technical and heavy going. Probably the most regarded source is a book by
Oded Goldreich [535] but this is pitched at the postgraduate maths student. A
less thorough but more readable introduction to randomness and algorithms
is in [564]. Current research at the theoretical end of cryptology is found at the
FOCS, STOC, Crypto, Eurocrypt and Asiacrypt conferences.
Four of the simple block cipher modes of operation (ECB, CBC, OFB and
CFB) date back to FIPS-81; their specification was reissued, with CTR mode
added, in 2001 as NIST Special Publication 800-38A [944]. The compound
modes of operation are described in subsequent papers in that series.
The history of cryptology is fascinating, and so many old problems keep on
recurring in modern guises that the security engineer should be familiar with
it. The standard work is Kahn [676]; there are also compilations of historical
articles from Cryptologia [363, 361, 362] as well as several books on the history
of cryptology in World War 2 [296, 677, 836, 1336]. The NSA Museum at Fort
George Meade, Md., is also worth a visit, as is the one at Bletchley Park in
England.
Finally, no chapter that introduces public key encryption would be complete
without a mention that, under the name of ‘non-secret encryption,’ it was first
discovered by James Ellis in about 1969. However, as Ellis worked for GCHQ
(Britain’s Government Communications Headquarters, the equivalent of the
NSA) his work remained classified. The RSA algorithm was then invented by
Clifford Cocks, and also kept secret. This story is told in [427]. One effect of
the secrecy was that their work was not used: although it was motivated by the
expense of Army key distribution, Britain’s Ministry of Defence did not start
building electronic key distribution systems for its main networks until 1992.
It should also be noted that the classified community did not pre-invent digital
signatures; they remain the achievement of Whit Diffie and Martin Hellman.

CHAPTER

6
Distributed Systems
You know you have a distributed system when the crash of a
computer you’ve never heard of stops you from
getting any work done.
— Leslie Lamport

6.1

Introduction

We’ve seen in the last few chapters how people can authenticate themselves to
systems (and systems can authenticate themselves to each other) using security
protocols; how access controls can be used to manage which principals can
perform what operations in a system; and some of the mechanics of how crypto
can be used to underpin access control in distributed systems. But there’s much
more to building a secure distributed system than just implementing access
controls, protocols and crypto. When systems become large, the scale-up
problems are not linear; there is often a qualitative change in complexity, and
some things that are trivial to deal with in a network of only a few machines
and principals (such as naming) suddenly become a big deal.
Over the last 40 years, computer science researchers have built many
distributed systems and studied issues such as concurrency, failure recovery
and naming. The theory is supplemented by a growing body of experience from
industry, commerce and government. These issues are central to the design
of effective secure systems but are often handled rather badly. I’ve already
described attacks on security protocols that can be seen as concurrency failures.
If we replicate data to make a system fault-tolerant then we may increase the
risk of a compromise of confidentiality. Finally, naming is a particularly
thorny problem. Many governments and organisations are trying to build
185

186

Chapter 6

■

Distributed Systems

larger, flatter namespaces — using identity cards to number citizens and using
RFID to number objects — and yet naming problems undermined attempts
during the 1990s to build useful public key infrastructures.

6.2

Concurrency

Processes are said to be concurrent if they run at the same time, and concurrency
gives rise to a number of well-studied problems. Processes may use old data;
they can make inconsistent updates; the order of updates may or may not
matter; the system might deadlock; the data in different systems might never
converge to consistent values; and when it’s important to make things happen
in the right order, or even to know the exact time, this can be harder than you
might think.
Systems are now rapidly becoming more concurrent. First, the scale of
online business has grown rapidly; Google may have started off with four
machines but now its server farms have hundreds of thousands. Second,
devices are becoming more complex; a luxury car can now contain over forty
different processors. Third, the components are also getting more complex:
the microprocessor in your PC may now have two, or even four, CPU cores,
and will soon have more, while the graphics card, disk controller and other
accessories all have their own processors too. On top of this, virtualization
technologies such as VMware and Xen may turn a handful of real CPUs into
hundreds or even thousands of virtual CPUs.
Programming concurrent systems is hard; and, unfortunately, most of the
textbook examples come from the relatively rarefied world of operating system
internals and thread management. But concurrency control is also a security
issue. Like access control, it exists in order to prevent users interfering with
each other, whether accidentally or on purpose. Also, concurrency problems
can occur at many levels in a system, from the hardware right up to the business
environment. In what follows, I provide a number of concrete examples of the
effects of concurrency on security. These are by no means exhaustive.

6.2.1

Using Old Data Versus Paying to Propagate State

I’ve already described two kinds of concurrency problem. First, there are
replay attacks on protocols, where an attacker manages to pass off out-ofdate credentials. Secondly, there are race conditions. I mentioned the ‘mkdir’
vulnerability from Unix, in which a privileged instruction that is executed in
two phases could be attacked halfway through the process by renaming an
object on which it acts. These problems have been around for a long time.
In one of the first multiuser operating systems, IBM’s OS/360, an attempt to

6.2 Concurrency

open a file caused it to be read and its permissions checked; if the user was
authorized to access it, it was read again. The user could arrange things so that
the file was altered in between [774].
These are examples of a time-of-check-to-time-of-use (TOCTTOU) attack. There
are systematic ways of finding such attacks in file systems [176], but as more
of our infrastructure becomes concurrent, attacks crop up at other levels
such as system calls in virtualised environments, which may require different
approaches. (I’ll discuss this specific case in detail in Chapter 18.) They also
appear at the level of business logic. Preventing them isn’t always economical,
as propagating changes in security state can be expensive.
For example, the banking industry manages lists of all hot credit cards
(whether stolen or abused) but there are millions of them worldwide, so it
isn’t possible to keep a complete hot card list in every merchant terminal, and
it would be too expensive to verify all transactions with the bank that issued
the card. Instead, there are multiple levels of stand-in processing. Terminals
are allowed to process transactions up to a certain limit (the floor limit) offline;
larger transactions need online verification with a local bank, which will know
about all the local hot cards plus foreign cards that are being actively abused;
above another limit there might be a reference to an organization such as VISA
with a larger international list; while the largest transactions might need a
reference to the card issuer. In effect, the only transactions that are checked
immediately before use are those that are local or large.
Credit card systems are interesting as the largest systems that manage
the global propagation of security state — which they do by assuming that
most events are local, of low value, or both. They taught us that revoking
compromised credentials quickly and on a global scale was expensive. In the
1990s, when people started to build infrastructures of public key certificates to
support everything from web shopping to corporate networks, there was a fear
that biggest cost would be revoking the credentials of principals who changed
address, changed job, had their private key hacked, or got fired. This turned
out not to be the case in general1 . Another aspect of the costs of revocation can
be seen in large web services, where it would be expensive to check a user’s
credentials against a database every time she visits any one of the service’s
thousands of machines. A common solution is to use cookies — giving the
user an encrypted credential that her browser automatically presents on each
visit. That way only the key has to be shared between the server farm’s many
machines. However, if revoking users quickly is important to the application,
some other method needs to be found to do this.
1
Frauds against web-based banking and shopping services don’t generally involve compromised
certificates. However, one application where revocation is a problem is the Deparatment of
Defense, which has issued 16 million certificates to military personnel since 1999 and now has
a list of 10 million revoked certificates that must be downloaded to all security servers every
day [878].

187

188

Chapter 6

6.2.2

■

Distributed Systems

Locking to Prevent Inconsistent Updates

When a number of people are working concurrently on a document, they may
use a version control system to ensure that only one person has write access at
any one time to any given part of it. This illustrates the importance of locking
as a way to manage contention for resources such as filesystems and to reduce
the likelihood of conflicting updates. Another mechanism is callback; a server
may keep a list of all those clients which rely on it for security state, and notify
them when the state changes.
Locking and callback also matter in secure distributed systems. Credit cards
again provide an example. If I own a hotel, and a customer presents a credit
card on checkin, I ask the card company for a pre-authorization which records
the fact that I will want to make a debit in the near future; I might register
a claim on ‘up to $500’ of her available credit. If the card is cancelled the
following day, her bank can call me and ask me to contact the police, or to
get her to pay cash. (My bank might or might not have guaranteed me the
money; it all depends on what sort of contract I’ve managed to negotiate
with it.) This is an example of the publish-register-notify model of how to do
robust authorization in distributed systems (of which there’s a more general
description in [105]).
Callback mechanisms don’t provide a universal solution, though. The credential issuer might not want to run a callback service, and the customer
might object on privacy grounds to the issuer being told all her comings and
goings. Consider passports as an example. In many countries, government
ID is required for many transactions, but governments won’t provide any
guarantee, and most citizens would object if the government kept a record
of every time an ID document was presented. Indeed, one of the frequent
objections to the British government’s proposal for biometric ID cards is that
checking citizens’ fingerprints against a database whenever they show their
ID would create an audit trail of all the places where the card was used.
In general, there is a distinction between those credentials whose use gives
rise to some obligation on the issuer, such as credit cards, and the others, such
as passports. Among the differences is the importance of the order in which
updates are made.

6.2.3

The Order of Updates

If two transactions arrive at the government’s bank account — say a credit of
$500,000 and a debit of $400,000 — then the order in which they are applied
may not matter much. But if they’re arriving at my bank account, the order will
have a huge effect on the outcome! In fact, the problem of deciding the order in
which transactions are applied has no clean solution. It’s closely related to the
problem of how to parallelize a computation, and much of the art of building

6.2 Concurrency

efficient distributed systems lies in arranging matters so that processes are
either simple sequential or completely parallel.
The usual algorithm in retail checking account systems is to batch the
transactions overnight and apply all the credits for each account before
applying all the debits. Inputs from devices such as ATMs and check sorters
are first batched up into journals before the overnight reconciliation. The
inevitable side-effect of this is that payments which bounce then have to be
reversed out — and in the case of ATM and other transactions where the cash
has already been dispensed, you can end up with customers borrowing money
without authorization. In practice, chains of failed payments terminate, though
in theory this isn’t necessarily so. Some interbank payment mechanisms are
moving to real time gross settlement in which transactions are booked in order
of arrival. The downside here is that the outcome can depend on network
vagaries. Some people thought this would limit the systemic risk that a nonterminating payment chain might bring down the world’s banking system,
but there is no real agreement on which practice is better. Credit cards operate
a mixture of the two strategies, with credit limits run in real time or near real
time (each authorization reduces the available credit limit) while settlement is
run just as in a checking account. The downside here is that by putting through
a large pre-authorization, a merchant can tie up your card.
The checking-account approach has recently been the subject of research in
the parallel systems community. The idea is that disconnected applications
propose tentative update transactions that are later applied to a master
copy. Various techniques can be used to avoid instability; mechanisms for
tentative update, such as with bank journals, are particularly important [553].
Application-level sanity checks are important; banks know roughly how much
they expect to pay each other each day to settle net payments, and large cash
flows get verified.
In other systems, the order in which transactions arrive is much less
important. Passports are a good example. Passport issuers only worry about
their creation and expiration dates, not the order in which visas are stamped
on them. (There are exceptions, such as the Arab countries that won’t let you
in if you have an Israeli stamp on your passport, but most pure identification
systems are stateless.)

6.2.4 Deadlock
Deadlock is another problem. Things may foul up because two systems are
each waiting for the other to move first. A famous exposition of deadlock is
the dining philosophers’ problem in which a number of philosophers are seated
round a table. There is a chopstick between each philosopher, who can only
eat when he can pick up the two chopsticks on either side. Deadlock can follow
if they all try to eat at once and each picks up (say) the chopstick on his right.

189

190

Chapter 6

■

Distributed Systems

This problem, and the algorithms that can be used to avoid it, are presented in
a classic paper by Dijkstra [388].
This can get horribly complex when you have multiple hierarchies of locks,
and they’re distributed across systems some of which fail (especially where
failures can mean that the locks aren’t reliable). There’s a lot written on the
problem in the distributed systems literature [104]. But it is not just a technical
matter; there are many Catch-22 situations in business processes. So long as
the process is manual, some fudge may be found to get round the catch, but
when it is implemented in software, this option may no longer be available.
Sometimes it isn’t possible to remove the fudge. In a well known business
problem — the battle of the forms — one company issues an order with its
own terms attached, another company accepts it subject to its own terms,
and trading proceeds without any agreement about whose conditions govern
the contract. The matter may only be resolved if something goes wrong
and the two companies end up in court; even then, one company’s terms
might specify an American court while the other’s specify a court in England.
This kind of problem looks set to get worse as trading becomes more electronic.

6.2.5

Non-Convergent State

When designing protocols that update the state of a distributed system, the
‘motherhood and apple pie’ is ACID — that transactions should be atomic,
consistent, isolated and durable. A transaction is atomic if you ‘do it all or not at
all’ — which makes it easier to recover the system after a failure. It is consistent
if some invariant is preserved, such as that the books must still balance. This
is common in banking systems, and is achieved by insisting that each credit
to one account is matched by an equal and opposite debit to another (I’ll
discuss this more in Chapter 10, ‘Banking and Bookkeeping’). Transactions are
isolated if they look the same to each other, that is, are serializable; and they
are durable if once done they can’t be undone.
These properties can be too much, or not enough, or both. On the one hand,
each of them can fail or be attacked in numerous obscure ways; on the other,
it’s often sufficient to design the system to be convergent. This means that, if the
transaction volume were to tail off, then eventually there would be consistent
state throughout [912]. Convergence is usually achieved using semantic tricks
such as timestamps and version numbers; this can often be enough where
transactions get appended to files rather than overwritten.
However, in real life, you also need ways to survive things that go wrong and
are not completely recoverable. The life of a security or audit manager can be
a constant battle against entropy: apparent deficits (and surpluses) are always
turning up, and sometimes simply can’t be explained. For example, different
national systems have different ideas of which fields in bank transaction
records are mandatory or optional, so payment gateways often have to

6.2 Concurrency

guess data in order to make things work. Sometimes they guess wrong;
and sometimes people see and exploit vulnerabilities which aren’t understood
until much later (if ever). In the end, things get fudged by adding a correction
factor, called something like ‘branch differences’, and setting a target for
keeping it below a certain annual threshold.
Durability is a subject of debate in transaction processing. The advent of
phishing and keylogging attacks has meant that some small proportion of
bank accounts will at any time be under the control of criminals; money gets
moved both from and through them. When an account compromise is detected,
the bank moves to freeze it and to reverse any payments that have recently
been made from it. The phishermen naturally try to move funds through
institutions, or jurisdictions, that don’t do transaction reversal, or do it at best
slowly and grudgingly [55]. This sets up a tension between the recoverability
and thus the resilience of the payment system on the one hand, and transaction
durability and finality on the other. The solution may lie at the application
level, namely charging customers a premium for irrevocable payments and
letting the market allocate the associated risks to the bank best able to price it.
The battle of the forms mentioned in the above section gives an example of
a distributed non-electronic system that doesn’t converge.
In military systems, there is the further problem of dealing with users who
request some data for which they don’t have a clearance. For example, someone
at a dockyard might ask the destination of a warship that’s actually on a secret
mission carrying arms to Iran. If she isn’t allowed to know this, the system
may conceal the ship’s real destination by making up a cover story. Search may
have to be handled differently from specific enquiries; the joining-up of
intelligence databases since 9/11 has forced system builders to start sending
clearances along with search queries, otherwise sorting the results became
unmanageable. This all raises difficult engineering problems, with potentially severe conflicts between atomicity, consistency, isolation and durability
(not to mention performance), which will be discussed at more length in
Chapter 8, ‘Multilevel Security’.

6.2.6

Secure Time

The final kind of concurrency problem with special interest to the security
engineer is the provision of accurate time. As authentication protocols such
as Kerberos can be attacked by inducing an error in the clock, it’s not enough
to simply trust a time source on the network. A few years ago, the worry
was a Cinderella attack: if a security critical program such as a firewall has a
license with a timelock in it, an attacker might wind your clock forward ‘and
cause your software to turn into a pumpkin’. Things have become more acute
since the arrival of operating systems such as Vista with hardware security
support, and of media players with built-in DRM; the concern now is that

191

192

Chapter 6

■

Distributed Systems

someone might do a large-scale service-denial attack by convincing millions
of machines that their owners had tampered with the clock, causing their files
to become inaccessible.
Anyway, there are several possible approaches to the provision of secure
time.
You could furnish every computer with a radio clock, but that can be
expensive, and radio clocks — even GPS — can be jammed if the opponent is serious.
There are clock synchronization protocols described in the research literature in which a number of clocks vote in a way that should make clock
failures and network delays apparent. Even though these are designed to
withstand random (rather than malicious) failure, they can no doubt be
hardened by having the messages digitally signed.
You can abandon absolute time and instead use Lamport time in which all
you care about is whether event A happened before event B, rather than
what date it is [766]. Using challenge-response rather than timestamps in
security protocols is an example of this; another is given by timestamping services that continually hash all documents presented to them into a
running total that’s published, and can thus provide proof that a certain
document existed by a certain date [572].
However, in most applications, you are likely to end up using the network
time protocol (NTP). This has a moderate amount of protection, with clock
voting and authentication of time servers. It is dependable enough for many
purposes.

6.3

Fault Tolerance and Failure Recovery

Failure recovery is often the most important aspect of security engineering,
yet it is one of the most neglected. For many years, most of the research papers
on computer security have dealt with confidentiality, and most of the rest
with authenticity and integrity; availability has been neglected. Yet the actual
expenditures of a typical bank are the other way round. Perhaps a third of all IT
costs go on availability and recovery mechanisms, such as hot standby processing sites and multiply redundant networks; a few percent more get invested
in integrity mechanisms such as internal audit; and an almost insignificant
amount goes on confidentiality mechanisms such as encryption boxes. As you
read through this book, you’ll see that many other applications, from burglar
alarms through electronic warfare to protecting a company from Internet-based
service denial attacks, are fundamentally about availability. Fault tolerance
and failure recovery are a huge part of the security engineer’s job.

6.3 Fault Tolerance and Failure Recovery

Classical fault tolerance is usually based on mechanisms such as logs and
locking, and is greatly complicated when it must withstand malicious attacks
on these mechanisms. Fault tolerance interacts with security in a number of
ways: the failure model, the nature of resilience, the location of redundancy
used to provide it, and defense against service denial attacks. I’ll use the
following definitions: a fault may cause an error, which is an incorrect state; this
may lead to a failure which is a deviation from the system’s specified behavior.
The resilience which we build into a system to tolerate faults and recover
from failures will have a number of components, such as fault detection, error
recovery and if necessary failure recovery. The meaning of mean-time-beforefailure (MTBF) and mean-time-to-repair (MTTR) should be obvious.

6.3.1

Failure Models

In order to decide what sort of resilience we need, we must know what sort
of attacks are expected. Much of this will come from an analysis of threats
specific to our system’s operating environment, but there are some general
issues that bear mentioning.

6.3.1.1

Byzantine Failure

First, the failures with which we are concerned may be normal or Byzantine.
The Byzantine fault model is inspired by the idea that there are n generals
defending Byzantium, t of whom have been bribed by the Turks to cause as
much confusion as possible in the command structure. The generals can pass
oral messages by courier, and the couriers are trustworthy, so each general can
exchange confidential and authentic communications with each other general
(we could imagine them encrypting and computing a MAC on each message).
What is the maximum number t of traitors which can be tolerated?
The key observation is that if we have only three generals, say Anthony,
Basil and Charalampos, and Anthony is the traitor, then he can tell Basil ‘let’s
attack’ and Charalampos ‘let’s retreat’. Basil can now say to Charalampos
‘Anthony says let’s attack’, but this doesn’t let Charalampos conclude that
Anthony’s the traitor. It could just as easily have been Basil; Anthony could
have said ‘let’s retreat’ to both of them, but Basil lied when he said ‘Anthony
says let’s attack’.
This beautiful insight is due to Leslie Lamport, Robert Shostak and
Marshall Pease, who proved that the problem has a solution if and only
if n ≥ 3t + 1 [767]. Of course, if the generals are able to sign their messages,
then no general dare say different things to two different colleagues. This illustrates the power of digital signatures in particular and of end-to-end security
mechanisms in general. Relying on third parties to introduce principals to each

193

194

Chapter 6

■

Distributed Systems

other or to process transactions between them can give great savings, but if the
third parties ever become untrustworthy then it can impose significant costs.
Another lesson is that if a component that fails (or can be induced to fail
by an opponent) gives the wrong answer rather than just no answer, then it’s
much harder to build a resilient system using it. This has recently become a
problem in avionics, leading to an emergency Airworthiness Directive in April
2005 that mandated a software upgrade for the Boeing 777, after one of these
planes suffered a ‘flight control outage’ [762].

6.3.1.2

Interaction with Fault Tolerance

We can constrain the failure rate in a number of ways. The two most obvious are
by using redundancy and fail-stop processors. The latter process error-correction
information along with data, and stop when an inconsistency is detected; for
example, bank transaction processing will typically stop if an out-of-balance
condition is detected after a processing task. The two may be combined; IBM’s
System/88 minicomputer had two disks, two buses and even two CPUs, each
of which would stop if it detected errors; the fail-stop CPUs were built by
having two CPUs on the same card and comparing their outputs. If they
disagreed the output went open-circuit, thus avoiding the Byzantine failure
problem.
In general distributed systems, either redundancy or fail-stop processing can
make a system more resilient, but their side effects are rather different. While
both mechanisms may help protect the integrity of data, a fail-stop processor
may be more vulnerable to service denial attacks, whereas redundancy can
make confidentiality harder to achieve. If I have multiple sites with backup
data, then confidentiality could be broken if any of them gets compromised;
and if I have some data that I have a duty to destroy, perhaps in response to a
court order, then purging it from multiple backup tapes can be a headache.
It is only a slight simplification to say that while replication provides
integrity and availability, tamper resistance provides confidentiality too. I’ll
return to this theme later. Indeed, the prevalence of replication in commercial
systems, and of tamper-resistance in military systems, echoes their differing
protection priorities.
However, there are traps for the unwary. In one case in which I was called
on as an expert, my client was arrested while using a credit card in a store,
accused of having a forged card, and beaten up by the police. He was adamant
that the card was genuine. Much later, we got the card examined by VISA
who confirmed that it was indeed genuine. What happened, as well as we
can reconstruct it, was this. Credit cards have two types of redundancy on
the magnetic strip — a simple checksum obtained by combining together
all the bytes on the track using exclusive-or, and a cryptographic checksum
which we’ll describe in detail later in section 10.5.2. The former is there to

6.3 Fault Tolerance and Failure Recovery

detect errors, and the latter to detect forgery. It appears that in this particular
case, the merchant’s card reader was out of alignment in such a way as to
cause an even number of bit errors which cancelled each other out by chance
in the simple checksum, while causing the crypto checksum to fail. The result
was a false alarm, and a major disruption in my client’s life.
Redundancy is hard enough to deal with in mechanical systems. For
example, training pilots to handle multi-engine aircraft involves drilling them
on engine failure procedures, first in the simulator and then in real aircraft
with an instructor. Novice pilots are in fact more likely to be killed by an
engine failure in a multi-engine plane than in a single; landing in the nearest
field is less hazardous for them than coping with suddenly asymmetric thrust.
The same goes for instrument failures; it doesn’t help to have three artificial
horizons in the cockpit if, under stress, you rely on the one that’s broken. Aircraft are much simpler than many modern information systems — yet there
are still regular air crashes when pilots fail to manage the redundancy that’s
supposed to keep them safe. All too often, system designers put in multiple
protection mechanisms and hope that things will be ‘all right on the night’.
This might be compared to strapping a 40-hour rookie pilot into a Learjet and
telling him to go play. It really isn’t good enough. Please bear the aircraft
analogy in mind if you have to design systems combining redundancy and
security!
The proper way to do things is to consider all the possible use cases and
abuse cases of a system, think through all the failure modes that can happen
by chance (or be maliciously induced), and work out how all the combinations
of alarms will be dealt with — and how, and by whom. Then write up your
safety and security case and have it evaluated by someone who knows what
they’re doing. I’ll have more to say on this later in the chapter on ‘System
Evaluation and Assurance’.
Even so, large-scale system failures very often show up dependencies that
the planners didn’t think of. For example, Britain suffered a fuel tanker
drivers’ strike in 2001, and some hospitals had to close because of staff
shortages. The government allocated petrol rations to doctors and nurses, but
not to schoolteachers. So schools closed, and nurses had to stay home to look
after their kids, and this closed hospitals too. We are becoming increasingly
dependent on each other, and this makes contingency planning harder.

6.3.2 What Is Resilience For?
When introducing redundancy or other resilience mechanisms into a system,
we need to be very clear about what they’re for. An important consideration
is whether the resilience is contained within a single organization.
In the first case, replication can be an internal feature of the server to make
it more trustworthy. AT&T built a system called Rampart in which a number

195

196

Chapter 6

■

Distributed Systems

of geographically distinct servers can perform a computation separately and
combine their results using threshold decryption and signature [1065]; the idea
is to use it for tasks like key management [1066]. IBM developed a variant on
this idea called Proactive Security, where keys are regularly flushed through
the system, regardless of whether an attack has been reported [597]. The idea
is to recover even from attackers who break into a server and then simply bide
their time until enough other servers have also been compromised. The trick
of building a secure ‘virtual server’ on top of a number of cheap off-the-shelf
machines has turned out to be attractive to people designing certification
authority services because it’s possible to have very robust evidence of attacks
on, or mistakes made by, one of the component servers [337]. It also appeals
to a number of navies, as critical resources can be spread around a ship in
multiple PCs and survive most kinds of damage that don’t sink it [489].
But often things are much more complicated. A server may have to protect
itself against malicious clients. A prudent bank, for example, will assume that
some of its customers would cheat it given the chance. Sometimes the problem
is the other way round, in that we have to rely on a number of services, none
of which is completely trustworthy. Since 9/11, for example, international
money-laundering controls have been tightened so that people opening bank
accounts are supposed to provide two different items that give evidence of
their name and address — such as a gas bill and a pay slip. (This causes serious
problems in Africa, where the poor also need banking services as part of their
path out of poverty, but may live in huts that don’t even have addresses, let
alone utilities [55].)
The direction of mistrust has an effect on protocol design. A server faced with
multiple untrustworthy clients, and a client relying on multiple servers that
may be incompetent, unavailable or malicious, will both wish to control the
flow of messages in a protocol in order to contain the effects of service denial.
So a client facing several unreliable servers may wish to use an authentication
protocol such as the Needham-Schroeder protocol I discussed in section 3.7.2;
then the fact that the client can use old server tickets is no longer a bug but
a feature. This idea can be applied to protocol design in general [1043]. It
provides us with another insight into why protocols may fail if the principal
responsible for the design, and the principal who carries the cost of fraud, are
different; and why designing systems for the real world in which everyone
(clients and servers) are unreliable and mutually suspicious, is hard.
At a still higher level, the emphasis might be on security renewability. Pay-TV
is a good example: secret keys and other subscriber management tools are typically kept in a cheap smartcard rather than in an expensive set-top box, so that
even if all the secret keys are compromised, the operator can recover by mailing new cards out to his subscribers. I’ll discuss in more detail in Chapter 22,
‘Copyright and Privacy Protection’.

6.3 Fault Tolerance and Failure Recovery

6.3.3 At What Level Is the Redundancy?
Systems may be made resilient against errors, attacks and equipment failures at
a number of levels. As with access control systems, these become progressively
more complex and less reliable as we go up to higher layers in the system.
Some computers have been built with redundancy at the hardware level,
such as the IBM System/88 I mentioned earlier. From the late 1980’s, these
machines were widely used in transaction processing tasks (eventually ordinary hardware became reliable enough that banks would not pay the premium
in capital cost and development effort to use non-standard hardware). Some
more modern systems achieve the same goal with standard hardware either at
the component level, using redundant arrays of inexpensive disks (‘RAID’ disks)
or at the system level by massively parallel server farms. But none of these
techniques provides a defense against faulty or malicious software, or against
an intruder who exploits such software.
At the next level up, there is process group redundancy. Here, we may run
multiple copies of a system on multiple servers in different locations, and
compare their outputs. This can stop the kind of attack in which the opponent
gets physical access to a machine and subverts it, whether by mechanical
destruction or by inserting unauthorized software, and destroys or alters
data. It can’t defend against attacks by authorized users or damage by bad
authorized software, which could simply order the deletion of a critical file.
The next level is backup. Here, we typically take a copy of the system (also
known as a checkpoint) at regular intervals. The backup copies are usually
kept on media that can’t be overwritten such as write-protected tapes or
DVDs. We may also keep journals of all the transactions applied between
checkpoints. In general, systems are kept recoverable by a transaction processing strategy of logging the incoming data, trying to do the transaction,
logging it again, and then checking to see whether it worked. Whatever the
detail, backup and recovery mechanisms not only enable us to recover from
physical asset destruction; they also ensure that if we do get an attack at
the logical level — such as a time bomb in our software which deletes our
customer database on a specific date — we have some hope of recovering.
They are not infallible though. The closest that any bank I know of came
to a catastrophic computer failure that would have closed its business was
when its mainframe software got progressively more tangled as time progressed, and it just wasn’t feasible to roll back processing several weeks and
try again.
Backup is not the same as fallback. A fallback system is typically a less capable
system to which processing reverts when the main system is unavailable. An
example is the use of manual imprinting machines to capture credit card
transactions from the card embossing when electronic terminals fail.

197

198

Chapter 6

■

Distributed Systems

Fallback systems are an example of redundancy in the application layer —
the highest layer we can put it. We might require that a transaction above
a certain limit be authorized by two members of staff, that an audit trail be
kept of all transactions, and a number of other things. We’ll discuss such
arrangements at greater length in the chapter on banking and bookkeeping.
It is important to realise that these are different mechanisms, which do
different things. Redundant disks won’t protect against a malicious programmer who deletes all your account files, and backups won’t stop him if rather
than just deleting files he writes code that slowly inserts more and more
errors. Neither will give much protection against attacks on data confidentiality. On the other hand, the best encryption in the world won’t help you
if your data processing center burns down. Real world recovery plans and
mechanisms can get fiendishly complex and involve a mixture of all of the
above.
The remarks that I made earlier about the difficulty of redundancy, and
the absolute need to plan and train for it properly, apply in spades to system
backup. When I was working in banking we reckoned that we could probably
get our backup system working within an hour or so of our main processing
centre being destroyed, but the tests we did were limited by the fact that we
didn’t want to risk processing during business hours. The most impressive
preparations I’ve ever seen were at a UK supermarket, which as a matter
of policy pulls the plug on its main processing centre once a year without
warning the operators. This is the only way they can be sure that the backup
arrangements actually work, and that the secondary processing centre really
cuts in within a minute or so. Bank tellers can keep serving customers for a
few hours with the systems down; but retailers with dead checkout lanes can’t
do that.

6.3.4

Service-Denial Attacks

One of the reasons we want security services to be fault-tolerant is to make
service-denial attacks less attractive, more difficult, or both. These attacks are
often used as part of a larger attack plan. For example, one might swamp a
host to take it temporarily offline, and then get another machine on the same
LAN (which had already been subverted) to assume its identity for a while.
Another possible attack is to take down a security server to force other servers
to use cached copies of credentials.
A powerful defense against service denial is to prevent the opponent
mounting a selective attack. If principals are anonymous — or at least there
is no name service which will tell the opponent where to attack — then he
may be ineffective. I’ll discuss this further in the context of burglar alarms and
electronic warfare.

6.3 Fault Tolerance and Failure Recovery

Where this isn’t possible, and the opponent knows where to attack, then there
are some types of service-denial attacks which can be stopped by redundancy
and resilience mechanisms, and others which can’t. For example, the TCP/IP
protocol has few effective mechanisms for hosts to protect themselves against
various network flooding attacks. An opponent can send a large number
of connection requests and prevent anyone else establishing a connection.
Defense against this kind of attack tends to involve moving your site to a
beefier hosting service with specialist packet-washing hardware — or tracing
and arresting the perpetrator.
Distributed denial-of-service (DDoS) attacks had been known to the research
community as a possibility for some years. They came to public notice when
they were used to bring down Panix, a New York ISP, for several days in 1996.
During the late 1990s they were occasionally used by script kiddies to take over
chat servers. In 2000, colleagues and I suggested dealing with the problem by
server replication [1366], and in 2001 I mentioned them in passing in the first
edition of this book. Over the following three years, small-time extortionists
started using DDoS attacks for blackmail. The modus operandi was to assemble
a botnet, a network of compromised PCs used as attack robots, which would
flood a target webserver with packet traffic until its owner paid them to desist.
Typical targets were online bookmakers, and amounts of $10,000–$50,000
were typically demanded to leave them alone. The typical bookie paid up
the first time this happened, but when the attacks persisted the first solution
was replication: operators moved their websites to hosting services such as
Akamai whose servers are so numerous (and so close to customers) that they
can shrug off anything that the average botnet could throw at them. In the
end, the blackmail problem was solved when the bookmakers met and agreed
not to pay any more blackmail money, and the Russian police were prodded
into arresting the gang responsible.
Finally, where a more vulnerable fallback system exists, a common technique
is to force its use by a service denial attack. The classic example is the use of
smartcards for bank payments in countries in Europe. Smartcards are generally
harder to forge than magnetic strip cards, but perhaps 1% of them fail every
year, thanks to static electricity and worn contacts. Also, foreign tourists still
use magnetic strip cards. So card payment systems have a fallback mode that
uses the magnetic strip. A typical attack nowadays is to use a false terminal, or
a bug inserted into the cable between a genuine terminal and a branch server,
to capture card details, and then write these details to the magnetic stripe of
a card whose chip has been destroyed (connecting 20V across the contacts
does the job nicely). In the same way, burglar alarms that rely on network
connections for the primary response and fall back to alarm bells may be very
vulnerable if the network can be interrupted by an attacker: now that online
alarms are the norm, few people pay attention any more to alarm bells.

199

200

Chapter 6

6.4

■

Distributed Systems

Naming

Naming is a minor if troublesome aspect of ordinary distributed systems, but
it becomes surprisingly hard in security engineering. The topical example in
the 1990s was the problem of what names to put on public key certificates. A
certificate that says simply ‘the person named Ross Anderson is allowed to
administer machine X’ is little use. Before the arrival of Internet search engines,
I was the only Ross Anderson I knew of; now I know of dozens of us. I am
also known by different names to dozens of different systems. Names exist in
contexts, and naming the principals in secure systems is becoming ever more
important and difficult.
Engineers observed then that using more names than you need to causes
unnecessary complexity. For example, A certificate that simply says ‘the bearer
of this certificate is allowed to administer machine X’ is a straightforward
bearer token, which we know how to deal with; but once my name is involved,
then presumably I have to present some kind of ID to prove who I am,
and the system acquires a further dependency. Worse, if my ID is compromised
the consequences could be extremely far-reaching.
Since 9/11 the terms of this debate have shifted somewhat, as many
governments have rushed to issue their citizens with ID cards. In order to
justify the expense and hassle, pressure is often put on commercial system
operators to place some reliance on government-issue ID where this was
previously not thought necessary. In the UK, for example, you can no longer
board a domestic flight using just the credit card with which you bought the
ticket, and you have to produce a passport or driving license to cash a check or
order a bank transfer for more than £1000. Such measures cause inconvenience
and introduce new failure modes into all sorts of systems.
No doubt this identity fixation will abate in time, as governments find
other things to scare their citizens with. However there is a second reason
that the world is moving towards larger, flatter name spaces: the move from
barcodes (which code a particular product) to RFID tags (which contain a
128-bit unique identifier that code a particular item.) This has ramifications
well beyond naming — into issues such as the interaction between product
security, supply-chain security and competition policy.
For now, it’s useful to go through what a generation of computer science
researchers have learned about naming in distributed systems.

6.4.1

The Distributed Systems View of Naming

During the last quarter of the twentieth century, the distributed systems
research community ran up against many naming problems. The basic algorithm used to bind names to addresses is known as rendezvous: the principal

6.4 Naming

exporting a name advertises it somewhere, and the principal seeking to
import and use it searches for it. Obvious examples include phone books, and
directories in file systems.
However, the distributed systems community soon realised that naming
can get fiendishly complex, and the lessons learned are set out in a classic
article by Needham [958]. I’ll summarize the main points, and look at which
of them apply to secure systems.
1. The function of names is to facilitate sharing. This continues to hold: my
bank account number exists in order to provide a convenient way of
sharing the information that I deposited money last week with the
teller from whom I am trying to withdraw money this week. In general, names are needed when the data to be shared is changeable. If I
only ever wished to withdraw exactly the same sum as I’d deposited, a
bearer deposit certificate would be fine. Conversely, names need not be
shared — or linked — where data will not be; there is no need to link
my bank account number to my telephone number unless I am going to
pay my phone bill from the account.
2. The naming information may not all be in one place, and so resolving names
brings all the general problems of a distributed system. This holds with
a vengeance. A link between a bank account and a phone number
assumes both of them will remain stable. So each system relies on the
other, and an attack on one can affect the other. In the days when
electronic banking was dial-up rather than web based, a bank which
identified its customers using calling line ID was vulnerable to attacks
on telephone systems (such as tapping into the distribution frame in
an apartment block, hacking a phone company computer, or bribing a phone company employee). Nowadays some banks are using
two-channel authorization to combat phishing — if you order a payment online you get a text message on your mobile phone saying ‘if
you want to pay $X to account Y, please enter the following four digit
code into your browser’. This is a bit tougher, as mobile phone traffic
is encrypted — but one weak link to watch for is the binding between
the customer and his phone number. If you let just anyone notify you of
a customer phone number change, you’ll be in trouble.
3. It is bad to assume that only so many names will be needed. The shortage of
IP addresses, which motivated the development of IP version 6 (IPv6),
is well enough discussed. What is less well known is that the most
expensive upgrade which the credit card industry ever had to make
was not Y2K remediation, but the move from thirteen digit credit card
numbers to sixteen. Issuers originally assumed that 13 digits would
be enough, but the system ended up with tens of thousands of banks
(many with dozens of products) so a six digit bank identification number

201

202

Chapter 6

■

Distributed Systems

(BIN number) was needed. Some card issuers have millions of
customers, so a nine digit account number is the norm. And there’s also
a check digit (a one-digit linear combination of the other digits which is
appended to detect errors).
4. Global names buy you less than you think. For example, the 128-bit address
in IPv6 can in theory enable every object in the universe to have a
unique name. However, for us to do business, a local name at my end
must be resolved into this unique name and back into a local name at
your end. Invoking a unique name in the middle may not buy us anything; it may even get in the way if the unique naming service takes
time, costs money, or occasionally fails (as it surely will). In fact, the
name service itself will usually have to be a distributed system, of
the same scale (and security level) as the system we’re trying to protect. So we can expect no silver bullets from this quarter. One reason
the banking industry was wary of initiatives such as SET that would
have given each customer a public key certificate on a key with which
they could sign payment instructions was that banks already have perfectly good names for their customers (account numbers). Adding an
extra name has the potential to add extra costs and failure modes.
5. Names imply commitments, so keep the scheme flexible enough to cope with
organizational changes. This sound principle was ignored in the design
of a UK government’s key management system for secure email [76].
There, principals’ private keys are generated by encrypting their names
under departmental master keys. So the frequent reorganizations meant
that the security infrastructure would have to be rebuilt each time —
and that money would have had to be spent solving many secondary
problems such as how people would access old material.
6. Names may double as access tickets, or capabilities. We have already seen
a number of examples of this in Chapters 2 and 3. In general, it’s a bad
idea to assume that today’s name won’t be tomorrow’s password or
capability — remember the Utrecht fraud we discussed in section 3.5.
(This is one of the arguments for making all names public keys — ‘keys
speak in cyberspace’ in Carl Ellison’s phrase — but we’ve already noted
the difficulties of linking keys with names.)
I’ve given a number of examples of how things go wrong when a name
starts being used as a password. But sometimes the roles of name and
password are ambiguous. In order to get entry to a car park I used to
use at the university, I had to speak my surname and parking badge
number into a microphone at the barrier. So if I say, ‘Anderson, 123’,
which of these is the password? In fact it was ‘Anderson’, as
anyone can walk through the car park and note down valid badge

6.4 Naming

numbers from the parking permits on the car windscreens. Another
example, from medical informatics, is a large database of medical
records kept by the UK government for research, where the name of the
patient has been replaced by their postcode and date of birth. Yet access
to many medical services requires the patient to give just these two
items to the receptionist to prove who they are. I will have more to say
on this later.
7. Things are made much simpler if an incorrect name is obvious. In standard
distributed systems, this enables us to take a liberal attitude to cacheing.
In payment systems, credit card numbers may be accepted while a terminal is offline so long as the credit card number appears valid (i.e., the
last digit is a proper check digit of the first fifteen) and it is not on the
hot card list. Certificates provide a higher-quality implementation of
the same basic concept.
It’s important where the name is checked. The credit card check digit
algorithm is deployed at the point of sale, so it is public. A further
check — the card verification value on the magnetic strip — is computed
with secret keys but can be checked at the issuing bank, the acquiring bank or even at a network switch (if one trusts these third parties
with the keys). This is more expensive, and still vulnerable to network
outages.
8. Consistency is hard, and is often fudged. If directories are replicated, then
you may find yourself unable to read, or to write, depending on whether too
many or too few directories are available. Naming consistency causes problems for e-commerce in a number of ways, of which perhaps the most
notorious is the bar code system. Although this is simple enough in
theory — with a unique numerical code for each product — in practice it can be a nightmare, as different manufacturers, distributors and
retailers attach quite different descriptions to the bar codes in their
databases. Thus a search for products by ‘Kellogg’s’ will throw up
quite different results depending on whether or not an apostrophe is
inserted, and this can cause great confusion in the supply chain. Proposals to fix this problem can be surprisingly complicated [618]. There
are also the issues of convergence discussed above; data might not be
consistent across a system, even in theory. There are also the problems of timeliness, such as whether a product has been recalled.
Now, many firms propose moving to RFID chips that contain a globally unique number: an item code rather than a product code. This may
move name resolution upstream; rather than the shop’s
computer recognising that the customer has presented a packet of
vitamin C at the checkout, it may go to the manufacturer to find this

203

204

Chapter 6

■

Distributed Systems

out. Manufacturers push for this on safety grounds; they can then be
sure that the product hasn’t passed its sell-by date and has not been
recalled. But this also increases their power over the supply chain;
they can detect and stop gray-market trading that would otherwise
undermine their ability to charge different prices in different towns.
9. Don’t get too smart. Phone numbers are much more robust than computer
addresses. Amen to that — but it’s too often ignored by secure system
designers. Bank account numbers are much easier to deal with than
the public-key certificates which were once believed both necessary
and sufficient to secure credit card payments online. I’ll discuss specific problems of public key infrastructures in section 21.4.5.7.
10. Some names are bound early, others not; and in general it is a bad thing to
bind early if you can avoid it. A prudent programmer will normally avoid
coding absolute addresses or filenames as that would make it hard to
upgrade or replace a machine. He will prefer to leave this to a configuration file or an external service such as DNS. (This is another reason
not to put addresses in names.) It is true that secure systems often
want stable and accountable names as any third-party service used
for last minute resolution could be a point of attack. However, designers should read the story of Netgear, who got their home routers to find
out the time using the Network Time Protocol from a server
at the University of Wisconsin-Madison. Their product was successful;
the university was swamped with hundreds of thousands of packets a
second. Netgear ended up paying them $375,000 to maintain the time
service for three years. Shortly afterwards, D-Link repeated the same
mistake [304].
So Needham’s ten principles for distributed naming apply fairly directly to
distributed secure systems.

6.4.2

What Else Goes Wrong

Needham’s principles, although very useful, are not sufficient. They were
designed for a world in which naming systems could be designed and
imposed at the system owner’s convenience. When we move from distributed
systems in the abstract to the reality of modern web-based (and interlinked)
service industries, there is still more to say.

6.4.2.1

Naming and Identity

The most obvious difference is that the principals in security protocols may be
known by many different kinds of name — a bank account number, a company
registration number, a personal name plus a date of birth or a postal address,

6.4 Naming

a telephone number, a passport number, a health service patient number, or a
userid on a computer system.
As I mentioned in the introductory chapter, a common mistake is to confuse
naming with identity. Identity is when two different names (or instances of the
same name) correspond to the same principal (this is known in the distributed
systems literature as an indirect name or symbolic link). The classic example
comes from the registration of title to real estate. It is very common that
someone who wishes to sell a house uses a different name than they did at
the time it was purchased: they might have changed their name on marriage,
or after a criminal conviction. Changes in name usage are also common. For
example, the DE Bell of the Bell-LaPadula system2 wrote his name ‘D. Elliot
Bell’ in 1973 on that paper; but he was always known as David, which is how
he now writes his name too. A land-registration system must cope with a lot
of identity issues like this.
The classic example of identity failure leading to compromise is check fraud.
Suppose I steal a high-value check made out to Mr Elliott Bell. I then open an
account in that name and cash it; banking law in both the USA and the UK
absolves the bank of liability so long as it pays the check into an account of the
same name. The modern procedure of asking people who open bank accounts
for two proofs of address, such as utility bills, probably makes the bad guys’
job easier; there are hundreds of utility companies, many of which provide
electronic copies of bills that are easy to alter. The pre-9/11 system, of taking
up personal references, may well have been better.
Moving to verifying government-issue photo-ID on account opening adds
to the mix statements such as ‘The Elliott Bell who owns bank account number
12345678 is the Elliott James Bell with passport number 98765432 and date
of birth 3/4/56’. This may be seen as a symbolic link between two separate
systems — the bank’s and the passport office’s. Note that the latter part of
this ‘identity’ encapsulates a further statement, which might be something
like ‘The US passport office’s file number 98765432 corresponds to the entry
in birth register for 3/4/56 of one Elliott James Bell’. In general, names may
involve several steps of recursion, and this gives attackers a choice of targets.
For example, a lot of passport fraud is pre-issue fraud: the bad guys apply
for passports in the names of genuine citizens who haven’t applied for a
passport already and for whom copies of birth certificates are easy enough to
obtain. Postmortem applications are also common. Linden Labs, the operators
of Second Life, introduced in late 2007 a scheme whereby you prove you’re
over 18 by providing the driver’s license number or social security number
of someone who is. Now a web search quickly pulls up such data for many
people, such as the rapper Tupac Amaru Shakur; and yes, Linden Labs did
accept Mr Shakur’s license number — even through the license is expired, and
2

I’ll discuss this in Chapter 8, ‘Multilevel Secure Systems’.

205

206

Chapter 6

■

Distributed Systems

he’s dead. Indeed, someone else managed to verify their age using Mohammed
Atta’s driver’s license [389].

6.4.2.2

Cultural Assumptions

The assumptions that underlie names often change from one country to
another. In the English-speaking world, people may generally use as many
names as they please; a name is simply what you are known by. But some
countries forbid the use of aliases, and others require them to be registered.
This can lead to some interesting scams: in at least one case, a British citizen
evaded pursuit by foreign tax authorities by changing his name. On a less
exalted plane, women who pursue academic careers and change their name
on marriage may wish to retain their former name for professional use, which
means that the name on their scientific papers is different from their name on
the payroll. This caused a row at my university which introduced a unified ID
card system, keyed to payroll names, without support for aliases.
In general, many of the really intractable problems arise when an attempt is
made to unify two local naming systems which turn out to have incompatible
assumptions. As electronics invades everyday life more and more, and systems
become linked up, conflicts can propagate and have unexpected remote effects.
For example, one of the lady professors in the dispute over our university card
was also a trustee of the British Library, which issues its own admission tickets
on the basis of the name on the holder’s home university library card.
Even human naming conventions are not uniform. Russians are known by
a forename, a patronymic and a surname; Icelanders have no surname but are
known instead by a given name followed by a patronymic if they are male
and a matronymic if they are female. This causes problems when they travel.
When US immigration comes across ‘Maria Trosttadóttir’ and learns that
‘Trosttadóttir’ isn’t a surname or even a patronymic, their standard practice
was to compel her to adopt as a surname a patronymic (say, ‘Carlsson’ if her
father was called Carl). This causes unnecessary offence.
The biggest cultural divide is often thought to be that between the English
speaking countries (where identity cards were long considered to be unacceptable on privacy grounds3 ) and the countries conquered by Napoleon or by the
Soviets, where identity cards are the norm.
There are further subtleties. I know Germans who have refused to believe
that a country could function at all without a proper system of population registration and ID cards, yet admit they are asked for their ID card only rarely (for
example, to open a bank account or get married). Their card number can’t be
used as a name, because it is a document number and changes every time a new
card is issued. A Swiss hotelier may be happy to register a German guest on
3

unless they’re called drivers’ licences or health service cards!

6.4 Naming

sight of an ID card rather than a credit card, but if he discovers some damage
after a German guest has checked out, he may be out of luck. And the British
passport office will issue a citizen with more than one passport at the same
time, if he says he needs them to make business trips to (say) Cuba and the
USA; so our Swiss hotelier, finding that a British guest has just left without
paying, can’t rely on the passport number to have him stopped at the airport.
There are many other hidden assumptions about the relationship between
governments and people’s names, and they vary from one country to another
in ways which can cause unexpected security failures.

6.4.2.3

Semantic Content of Names

Another hazard arises on changing from one type of name to another without
adequate background research. A bank got sued after they moved from storing
customer data by account number to storing it by name and address. They
wanted to target junk mail more accurately, so they wrote a program to link
up all the accounts operated by each of their customers. The effect for one poor
customer was that the bank statement for the account he maintained for his
mistress got sent to his wife, who divorced him.
Sometimes naming is simple, but sometimes it merely appears to be. For
example, when I got a monthly ticket for the local swimming baths, the cashier
simply took the top card off a pile, swiped it through a reader to tell the system
it was now live, and gave it to me. I had been assigned a random name — the
serial number on the card. Many US road toll systems work in much the same
way. Sometimes a random, anonymous name can add commercial value. In
Hong Kong, toll tokens for the Aberdeen tunnel could be bought for cash, or at
a discount in the form of a refillable card. In the run-up to the transfer of power
from Britain to Beijing, many people preferred to pay extra for the less traceable
version as they were worried about surveillance by the new government.
Semantics of names can change. I once got a hardware store loyalty card with
a random account number (and no credit checks). I was offered the chance to
change this into a bank card after the store was taken over by the supermarket
and the supermarket started a bank. (This appears to have ignored money
laundering regulations that all new bank customers must be subjected to due
diligence.)
Assigning bank account numbers to customers might have seemed unproblematic — but as the above examples show, systems may start to construct
assumptions about relationships between names that are misleading and
dangerous.

6.4.2.4

Uniqueness of Names

Human names evolved when we lived in small communities. We started off
with just forenames, but by the late Middle Ages the growth of travel led

207

208

Chapter 6

■

Distributed Systems

governments to bully people into adopting surnames. That process took a
century or so, and was linked with the introduction of paper into Europe
as a lower-cost and more tamper-resistant replacement for parchment; paper
enabled the badges, seals and other bearer tokens, which people had previously used for road tolls and the like, to be replaced with letters that mentioned
their names.
The mass movement of people, business and administration to the Internet
in the decade after 1995 has been too fast to allow any such social adaptation.
There are now many more people (and systems) online than we are used
to dealing with. As I remarked at the beginning of this section, I used to
be the only Ross Anderson I knew of, but thanks to search engines, I now
know dozens of us. Some of us work in fields I’ve also worked in, such
as software engineering and electric power distribution; the fact that I’m
www.ross-anderson.com and ross.anderson@iee.org is down to luck — I
got there first. (Even so, rjanderson@iee.org is somebody else.) So even
the combination of a relatively rare name and a specialized profession is
still ambiguous. Another way of putting this is that ‘traditional usernames,
although old-school-geeky, don’t scale well to the modern Internet’ [21].
Sometimes system designers are tempted to solve the uniqueness problem
by just giving everything and everyone a number. This is very common in
transaction processing, but it can lead to interesting failures if you don’t put
the uniqueness in the right place. A UK bank wanted to send £20m overseas,
but the operator typed in £10m by mistake. To correct the error, a second
payment of £10m was ordered. However, the sending bank’s system took
the transaction sequence number from the paperwork used to authorise it.
Two payments were sent to SWIFT with the same date, payee, amount and
sequence number — so the second was discarded as a duplicate [218].

6.4.2.5

Stability of Names and Addresses

Many names include some kind of address, yet addresses change. About a
quarter of Cambridge phone book addresses change every year; with email,
the turnover is probably higher. A project to develop a directory of people who
use encrypted email, together with their keys, found that the main cause of
changed entries was changes of email address [67]. (Some people had assumed
it would be the loss or theft of keys; the contribution from this source was
precisely zero.)
A serious problem could arise with IPv6. The security community assumes
that v6 IP addresses will be stable, so that public key certificates can bind
principals of various kinds to them. All sorts of mechanisms have been
proposed to map real world names, addresses and even document content
indelibly and eternally on to 128 bit strings (see, for example, [573]). The data
communications community, on the other hand, assumes that IPv6 addresses

6.4 Naming

will change regularly. The more significant bits will change to accommodate
more efficient routing algorithms, while the less significant bits will be used to
manage local networks. These assumptions can’t both be right.
Distributed systems pioneers considered it a bad thing to put addresses in
names [912]. But in general, there can be multiple layers of abstraction with
some of the address information at each layer forming part of the name at the
layer above. Also, whether a namespace is better flat depends on the application. Often people end up with different names at the departmental and organizational level (such as rja14@cam.ac.uk and ross.anderson@cl.cam.ac.uk
in my own case). So a clean demarcation between names and addresses is not
always possible.
Authorizations have many (but not all) of the properties of addresses. Kent’s
Law tells designers that if a credential contains a list of what it may be used
for, then the more things are on this list the shorter its period of usefulness. A
similar problem besets systems where names are composite. For example, some
online businesses recognize me by the combination of email address and credit
card number. This is clearly bad practice. Quite apart from the fact that I have
several email addresses, I have several credit cards. The one I use will depend
on which of them is currently offering the best service or the biggest bribes.
There are many good reasons to use pseudonyms. It’s certainly sensible
for children and young people to use online names that aren’t easily linkable
to their real names. This is often advocated as a child-protection measure,
although the number of children abducted and murdered by strangers in
developed countries remains happily low and stable at about 1 per 10,000,000
population per year. A more serious reason is that when you go for your first
job on leaving college aged 22, or for a CEO’s job at 45, you don’t want Google
to turn up all your teenage rants. Many people also change email addresses
from time to time to escape spam; I give a different email address to every
website where I shop. Of course, there are police and other agencies that
would prefer people not to use pseudonyms, and this takes us into the whole
question of traceability online, which I’ll discuss in Part II.

6.4.2.6

Adding Social Context to Naming

The rapid growth recently of social network sites such as Facebook points to
a more human and scaleable way of managing naming. Facebook does not
give me a visible username: I use my own name, and build my context by
having links to a few dozen friends. (Although each profile does have a unique
number, this does not appear in the page itself, just in URLs.) This fixes the
uniqueness problem — Facebook can have as many Ross Andersons as care
to turn up — and the stability problem (though at the cost of locking me into
Facebook if I try to use it for everything).

209

210

Chapter 6

■

Distributed Systems

Distributed systems folks had argued for some time that no naming system
can be simultaneously globally unique, decentralized, and human-meaningful.
It can only have two of those attributes (Zooko’s triangle) [21]. In the past,
engineers tanded to look for naming systems that were unique and meaningful,
like URLs, or unique and decentralised, as with public-key certificates4 . The
innovation from sites like Facebook is to show on a really large scale that
naming doesn’t have to be unique at all. We can use social context to build
systems that are both decentralised and meaningful — which is just what our
brains evolved to cope with.

6.4.2.7

Restrictions on the Use of Names

The interaction between naming and society brings us to a further problem:
some names may be used only in restricted circumstances. This may be laid
down by law, as with the US Social Security Number (SSN) and its equivalents
in many European countries. Sometimes it is a matter of marketing. I would
rather not give out my residential address (or my home phone number) when
shopping online, and I avoid businesses that demand them.
Restricted naming systems interact in unexpected ways. For example, it’s
fairly common for hospitals to use a patient number as an index to medical
record databases, as this may allow researchers to use pseudonymous records
for some limited purposes without much further processing. This causes
problems when a merger of health maintenance organizations, or a new policy
directive in a national health service, forces the hospital to introduce uniform
names. In the UK, for example, the merger of two records databases — one of
which used patient names while the other was pseudonymous — has raised
the prospect of legal challenges to processing on privacy grounds.
Finally, when we come to law and policy, the definition of a name turns
out to be unexpectedly tricky. Regulations that allow police to collect communications data — that is, a record of who called whom and when — are
often very much more lax than the regulations governing phone tapping; in
many countries, police can get this data just by asking the phone company.
There was an acrimonious public debate in the UK about whether this enables
them to harvest the URLs which people use to fetch web pages. URLs often
have embedded in them data such as the parameters passed to search engines.
Clearly there are policemen who would like a list of everyone who hit a URL
such as http://www.google.com/search?q=cannabis+cultivation+UK; just as
clearly, many people would consider such large-scale trawling to be an unacceptable invasion of privacy. The police argued that if they were limited to
4
Carl Ellison, Butler Lampson and Ron Rivest went so far as to propose the SPKI/SDSI certificate
system in which naming would be relative, rather than fixed with respect to central authority.
The PGP web of trust worked informally in the same way.

6.5 Summary

monitoring IP addresses, they could have difficulties tracing criminals who
use transient IP addresses. In the end, Parliament resolved the debate when it
passed the Regulation of Investigatory Powers Act in 2000: the police just get
the identity of the machine under the laxer regime for communications data.

6.4.3 Types of Name
The complexity of naming appears at all levels — organisational, technical
and political. I noted in the introduction that names can refer not just to
persons (and machines acting on their behalf), but also to organizations, roles
(‘the officer of the watch’), groups, and compound constructions: principal in
role — Alice as manager; delegation — Alice for Bob; conjunction — Alice and
Bob. Conjunction often expresses implicit access rules: ‘Alice acting as branch
manager plus Bob as a member of the group of branch accountants’.
That’s only the beginning. Names also apply to services (such as NFS, or
a public key infrastructure) and channels (which might mean wires, ports,
or crypto keys). The same name might refer to different roles: ‘Alice as a
computer game player’ ought to have less privilege than ‘Alice the system
administrator’. The usual abstraction used in the security literature is to treat
them as different principals. This all means that there’s no easy mapping
between names and principals.
Finally, there are functional tensions which come from the underlying
business processes rather than from system design. Businesses mainly want
to get paid, while governments want to identify people uniquely. In effect,
business wants a credit card number while government wants a passport
number. Building systems which try to be both — as many governments are
trying to encourage — is a tar-pit. There are many semantic differences. You
can show your passport to a million people, if you wish, but you had better
not try that with a credit card. Banks want to open accounts for anyone who
turns up with some money; governments want them to verify people’s identity
carefully in order to discourage money laundering. The list is a long one.

6.5

Summary

Many secure distributed systems have incurred huge costs, or developed
serious vulnerabilities, because their designers ignored the basic lessons of
how to build (and how not to build) distributed systems. Most of these lessons
are still valid, and there are more to add.
A large number of security breaches are concurrency failures of one kind or
another; systems use old data, make updates inconsistently or in the wrong
order, or assume that data are consistent when they aren’t and can’t be.
Knowing the right time is harder than it seems.

211

212

Chapter 6

■

Distributed Systems

Fault tolerance and failure recovery are critical. Providing the ability to
recover from security failures, and random physical disasters, is the main
purpose of the protection budget for many organisations. At a more technical
level, there are significant interactions between protection and resilience mechanisms. Byzantine failure — where defective processes conspire, rather than
failing randomly — is an issue, and interacts with our choice of cryptographic
tools. There are many different flavors of redundancy, and we have to use the
right combination. We need to protect not just against failures and attempted
manipulation, but also against deliberate attempts to deny service which may
often be part of larger attack plans.
Many problems also arise from trying to make a name do too much, or
making assumptions about it which don’t hold outside of one particular
system, or culture, or jurisdiction. For example, it should be possible to revoke
a user’s access to a system by cancelling their user name without getting sued
on account of other functions being revoked. The simplest solution is often to
assign each principal a unique identifier used for no other purpose, such as a
bank account number or a system logon name. But many problems arise when
merging two systems that use naming schemes that are incompatible for some
reason. Sometimes this merging can even happen by accident — an example
being when two systems use a common combination such as ‘name plus date
of birth’ to track individuals, but in different ways.

Research Problems
In the research community, secure distributed systems tend to have been discussed as a side issue by experts on communications protocols and operating
systems, rather than as a discipline in its own right. So it is a relatively open
field, and one still holds much promise.
There are many technical issues which I’ve touched on in this chapter, such
as how we design secure time protocols and the complexities of naming. But
perhaps the most important research problem is to work out how to design
systems that are resilient in the face of malice, that degrade gracefully, and
whose security can be recovered simply once the attack is past. This may mean
revisiting the definition of convergent applications. Under what conditions
can one recover neatly from corrupt security state?
What lessons do we need to learn from the onset of phishing and keylogging
attacks on electronic banking, which mean that at any given time a small (but
nonzero) proportion of customer accounts will be under criminal control?
Do we have to rework recovery (which in its classic form explores how
to rebuild databases from backup tapes) into resilience, and if so how do
we handle the tensions with the classic notions of atomicity, consistency,

Further Reading

isolation and durability as the keys to convergence in distributed systems?
What interactions can there be between resilience mechanisms and the various
protection technologies? In what respects should the protection and resilience
mechanisms be aligned, and in what respects should they be separated? What
other pieces are missing from the jigsaw?

Further Reading
There are many books on distributed systems; I’ve found Mullender [912] to
be helpful and thought-provoking for graduate students, while the textbook
we recommend to our undergraduates by Bacon [104] is also worth reading.
Geraint Price has a survey of the literature on the interaction between fault
tolerance and security [1043]. The research literature on concurrency, such as
the SIGMOD conferences, has occasional gems. There is also a 2003 report
from the U.S. National Research Council, ‘Who Goes There? Authentication
Through the Lens of Privacy’ which discusses the tradeoffs between authentication and privacy, and how they tend to scale poorly [710].

213

CHAPTER

7

Economics
The great fortunes of the information age lie in the hands of
companies that have established proprietary
architectures that are used by a
large installed base of
locked-in customers.
— Carl Shapiro and Hal Varian
There are two things I am sure of after all these years: there is
a growing societal need for high assurance software, and
market forces are never going to provide it.
— Earl Boebert
If you try to buck the markets, then the markets will buck you.
— Margaret Thatcher

7.1

Introduction

The economics of information security has recently become a thriving and
fast-moving discipline. We started to realise round about 2000 that many
security system failures weren’t due to technical errors so much as to wrong
incentives: the classic case is where the people who guard a system are not
the people who suffer when it fails. Indeed, security mechanisms are often
designed quite deliberately to shift liability, which often leads to trouble.
Economics has always been important to engineering, at the raw level of
cost accounting; a good engineer was one who could build a bridge safely
with a thousand tons of concrete when everyone else used two thousand
tons. But the perverse incentives that arise in complex systems with multiple
owners make economic questions both more important and more subtle
215

216

Chapter 7

■

Economics

for the security engineer. Truly global-scale systems like the Internet arise
from the actions of millions of independent principals with divergent interests;
we hope that reasonable global outcomes will result from selfish local actions.
In general, people won’t do something unless they have an incentive to.
Markets are often the best guide we have to what sort of mechanisms work, or
fail. Markets also fail; the computer industry has been dogged by monopolies
since its earliest days. The reasons for this are now understood, and their
interaction with security is starting to be. When someone asks ‘Why is Microsoft
software insecure?’ we can now give a principled answer rather than simply
cursing Redmond as a form of bad weather.
The new field of security economics provides valuable insights not just
into ‘security’ topics such as privacy, bugs, spam, and phishing, but into
more general areas such as system dependability. For example, what’s the
optimal balance of effort by programmers and testers? (For the answer, see
section 7.5.1 below.) It also enables us to analyse the policy problems that
security technology increasingly throws up — on issues like digital rights
management. Where protection mechanisms are used by the system designer
to control the owner of a machine, rather than to protect her against outside
enemies, questions of competition policy and consumer rights follow, which
economics provides the language to discuss. There are also questions of the
balance between public and private action. Network insecurity is somewhat
like air pollution or congestion, in that people who connect insecure machines
to the Internet do not bear the full consequences of their actions. So how much
of the protection effort can (or should) be left to individuals, and how much
should be borne by vendors, regulators or the police?

7.2

Classical Economics

Modern economics is an enormous field covering many different aspects of
human behaviour. The parts of it that (so far) have found application in security
are largely drawn from microeconomics and game theory. I’ll discuss game
theory in the next section; here, I’ll give a helicopter tour of the most relevant
ideas from microeconomics. The object of the exercise is not to provide a tutorial
on economics, or even on information economics — for that, I recommend you
read Carl Shapiro and Hal Varian’s book ‘Information Rules’ [1159] — but to
familiarise you with the essential terminology and ideas, so we can move on
to discuss security economics.
The modern subject started in the 18th century when the industrial revolution and growing trade changed the world, and people wanted to understand
why. In 1776, Adam Smith’s classic ‘The Wealth of Nations’ [1192] provided a
first draft: he explained how rational self-interest in a free market economy
leads to economic wellbeing. Specialisation leads to productivity gains at
all levels from a small factory to international trade, and the self-interested

7.2 Classical Economics

striving of many individuals and firms drives progress, as people must produce something others value to survive in a competitive market. In his famous
phrase, ‘It is not from the benevolence of the butcher, the brewer, or the baker,
that we can expect our dinner, but from their regard to their own interest’.
These ideas were refined by nineteenth-century economists; David Ricardo
clarified and strengthened Smith’s arguments in favour of free trade, Stanley
Jevons, Léon Walras and Carl Menger built detailed models of supply and
demand, and by the end of the century Alfred Marshall had combined models
of supply and demand in markets for goods, labour and capital into an
overarching ‘classical’ model in which, at equilibrium, all the excess profits
would be competed away and the economy would be functioning efficiently.
By 1948, Kenneth Arrow and Gérard Debreu had put this on a rigorous
mathematical foundation by proving that markets give efficient outcomes,
subject to certain conditions. Much of the interest in economics — especially
to the computer industry, and to security folks in particular — comes from the
circumstances in which these conditions aren’t met, giving rise to monopolies
and other problems.

7.2.1 Monopoly
A rapid way into the subject is to consider a simple textbook case of monopoly.
Suppose we have a market for apartments in a university town, and the
students have different incomes. We might have one rich student able to pay
$4000 a month, maybe 300 people willing to pay $2000 a month, and (to give
us round numbers) 1000 prepared to pay $1000 a month. That gives us the
demand curve shown in Figure 7.1 below.
price

$4000 pm A

supply
$1400 pm
$1000 pm

C
E

B
D
demand
F
G
800 1000

Figure 7.1: The market for apartments

apartments

217

218

Chapter 7

■

Economics

So if there are 1000 apartments being let by many competing landlords,
the market-clearing price will be at the intersection of the demand curve
with the vertical supply curve, namely $1000. But suppose the market is
rigged — say the landlords have set up a cartel, or the university makes
its students rent through a tied agency. For simplicity let’s assume a single
monopolist landlord. He examines the demand curve, and notices that if he
rents out only 800 apartments, he can get $1400 per month for each of them.
Now 800 times $1400 is $1,120,000 per month, which is more than the million
dollars a month he’ll make from the market price at $1000. (Economists would
say that his ‘revenue box’ is CBFO rather than EDGO.) So he sets an artificially
high price, and 200 apartments remain empty.
This is clearly inefficient, and the Italian economist Vilfredo Pareto invented
a neat way to formalise this. A Pareto improvement is any change that would
make some people better off without making anyone else worse off, and an
allocation is Pareto efficient if there isn’t any Pareto improvement available.
Here, the allocation is not efficient, as the monopolist could rent out one empty
apartment to anyone at a lower price, making both him and them better off.
Now Pareto efficiency is a rather weak criterion; both perfect communism
(everyone gets the same income) and perfect dictatorship (the President gets
all the income) are Pareto-efficient. In neither case can you make anyone better
off without making someone worse off! Yet the simple monopoly described
here is not efficient even in this very weak sense.
So what can the monopolist do? There is one possibility — if he can charge
everyone a different price, then he can set each student’s rent at exactly what
they are prepared to pay. We call such a landlord a discriminating monopolist; he
charges the rich student exactly $4000, and so on down to the 1000th student
whom he charges exactly $1000. The same students get apartments as before,
yet almost all of them are worse off. The rich student loses $3000, money that
he was prepared to pay but previously didn’t have to; economists refer to this
money he saved as surplus. In effect, the discriminating monopolist manages
to extract all the consumer surplus.
Merchants have tried to price-discriminate since antiquity. The carpet seller
in Damascus who offers to ‘make a very special price, just for you’ is playing
this game, as is Microsoft in offering seven different versions of Vista at
different price points, and an airline in selling first, business and cattle class
seats. The extent to which firms can do this depends on a number of factors,
principally their market power and the amount of information they have.
Market power is a measure of how close a merchant is to being a monopolist;
under monopoly the merchant is a price setter while under perfect competition
he simply has to accept whatever price the market establishes (he is a price
taker). Technology tends to increase market power while reducing the cost of
information about customers at the same time, and this combination is one
of the main factors eroding privacy in the modern world.

7.2 Classical Economics

7.2.2 Public Goods
A second type of market failure occurs when everyone gets the same quantity
of some good, whether they want it or not. Classic examples are air quality,
national defense and scientific research. Economists call these public goods, and
the formal definition is that they are goods which are non-rivalrous (my using
them doesn’t mean there’s less available for you) and non-excludable (there’s
no practical way to exclude people from consuming them). Uncoordinated
markets are generally unable to provide public goods in socially optimal
quantities.
Public goods may be supplied by governments directly, as in the case of
national defense, or by using indirect mechanisms to coordinate markets. The
classic example is laws on patents and copyrights to encourage people to
produce inventions, literary works and musical compositions by giving them
a temporary monopoly — by making the goods in question excludable for a
limited period of time. Very often, public goods are provided by some mix
of public and private action; scientific research is done in universities that
get some public subsidy, earn some income from student fees, and get some
research contracts from industry (where industry may get patents on the useful
inventions while the underlying scientific research gets published for all to
use). The mix can be controversial; the debate on global warming sets people
who want direct government action in the form of a ‘carbon tax’ (which would
be simple and easy to enforce) against others who want a ‘cap and trade’
system whereby firms and countries can trade licenses to emit carbon (which
in a perfect world would cause emission reductions by the firms who could do
so most cheaply, but which might well be more open to abuse and evasion).
The importance of this for us is that many aspects of security are public
goods. I do not have an anti-aircraft gun on the roof of my house; air-defense
threats come from a small number of actors, and are most efficiently dealt with
by government action. So what about Internet security? Certainly there are
strong externalities involved, and people who connect insecure machines to the
Internet end up dumping costs on others, just like people who burn polluting
coal fires. So what should we do about it? One might imagine a government
tax on vulnerabilities, with rewards paid to researchers who discover them
and larger fines imposed on the firms whose software contained them. Again,
one of the early papers on security economics suggested a vulnerability capand-trade system; vendors who could not be bothered to make their software
secure could buy permits from other vendors who were making the effort
to tighten up their products [256]. (Both arrangements would be resisted by
the free software community!) But is air pollution the right analogy — or air
defense?
Threats such as viruses and spam used to come from a large number of
small actors, but since about 2004 we’ve seen a lot of consolidation as malware

219

220

Chapter 7

■

Economics

writers and users have become commercial. By 2007, the number of serious
spammers had dropped to the point that ISPs see significant fluctuations in
overall spam volumes as the big spammers run particular campaigns — there
is no law of large numbers operating any more [305]. This suggests a different
and perhaps more centralised strategy. If our air-defense threat in 1987 was
mainly the Russian airforce, and our cyber-defense threat in 2007 is mainly
from a small number of Russian gangs, and they are imposing large costs
on US and European Internet users and companies, then state action may be
needed now as it was then. Instead of telling us to buy anti-virus software,
our governments could be putting pressure on the Russians to round up and
jail their cyber-gangsters. I’ll discuss this in greater detail in Part III; for now,
it should be clear that concepts such as ‘monopoly’ and ‘public goods’ are
important to the security engineer — and indeed to everyone who works in
IT. Just think of the two operating systems that dominate the world’s desktops
and server farms: Windows is a monopoly, while the common Unix systems
(Linux and OpenBSD) are public goods maintained by volunteers. Why should
this be so? Why are markets for information goods and services so odd?

7.3

Information Economics

One of the insights from the nineteenth-century economists Jevons and Menger
is that the price of a good, at equilibrium, is the marginal cost of production.
When coal cost nine shillings a ton in 1870, that didn’t mean that every mine
dug coal at this price, merely that the marginal producers — those who were
only just managing to stay in business — could sell at that price. If the price
went down, these mines would close; if it went up, other, even more marginal
mines, would open. That’s how supply responded to changes in demand.

7.3.1

The Price of Information

So in a competitive equilibrium, the price of information should be its marginal
cost of production. But that is almost zero! This explains why there is so much
information available for free in the Internet; zero is its fair price. If two
or more suppliers compete to offer an operating system, or a map, or an
encyclopaedia, that they can duplicate for no cost, then the incentive will be
for them to keep on cutting their prices without limit. This is what happened
with encyclopaedias; the Britannica used to cost $1,600 for 32 volumes; then
Microsoft brought out Encarta for $49.95, forcing Britannica to produce a
cheap CD edition; and now we have Wikipedia for free [1159]. One firm after
another has had to move to a business model in which the goods are given
away free, and the money comes from advertising or in some parallel market.

7.3 Information Economics

Linux companies give away an operating system, and make their money from
support; many Linux developers give their time free to the project while at
college, as their contribution strengthens their CV and helps them get a good
job when they graduate.
Many other industries with high fixed costs and low marginal costs moved
to an advertising or service model; think terrestrial TV. Others have moved in
this direction: most newspapers made most of their money from advertising,
and so have had little difficulty moving to free online editions plus paid paper
editions, all putting the lucrative ads in front of eyeballs. Yet other industries,
such as airlines and hotels, tended instead to become monopolists who try to
dominate particular routes or areas and to charge different prices to different
customers.
So what other characteristics of the information goods and services industries are particularly important?
1. There are often network externalities, whereby the value of a network
grows more than linearly in the number of users. For example, the more
people used fax machines in the 1980s, the more useful they became,
until round about 1985 fax machines suddenly took off; every business
needed one. Much the same happened with email in about 1995. Network
effects also apply to services more generally: anyone wanting to auction
some goods will usually go to the largest auction house, as it will attract
more bidders. They also apply to software: firms develop software for
Windows so they will have access to more users than they would if they
developed for Linux or Mac, and users for their part prefer Windows
because there’s more software for it. (This is called a two-sided market.)
2. There is often technical lock-in stemming from interoperability. Once a
software firm is committed to using Windows as a platform for its product, it can be expensive to change; for users, too, changing platforms can
be expensive. They have to buy new software, convert files (if they can),
and retrain themselves.
These features separately can lead to industries with dominant firms;
together, they are even more likely to. If users simply want to be compatible with other users (and software vendors) then they will logically buy from
the vendor they expect to win the biggest market share.

7.3.2 The Value of Lock-In
There is an interesting result, due to Shapiro and Varian: that the value of
a software company is the total lock-in (due to both technical and network
effects) of all its customers [1159]. To see how this might work, consider a firm
with 100 staff each using Office, for which it has paid $500 per copy. It could

221

222

Chapter 7

■

Economics

save this $50,000 by moving to OpenOffice, so if the costs of installing this
product, retraining its staff, converting files and so on — in other words the
total switching costs — were less than $50,000, it would switch. But if the costs
of switching were more than $50,000, then Microsoft would put up its prices.
Technical lock-in existed before, but the move to the digital economy has
made it much more significant. If you own a Volvo car, you are locked in to
Volvo spares and accessories; but when you’re fed up with it you can always
trade it in for a Mercedes. But if you own an Apple Mac, you’ll have Mac
software, a Mac printer, and quite possibly thousands of music tracks that
you’ve ripped to iTunes. You’d also have to learn to use different commands
and interfaces. Moving to Windows would be much more painful than just
shelling out $700 for a new laptop. You’d have to retrain yourself; you’d have to
throw away Office for Mac and buy Office for Windows. And if you’d bought
a lot of music tracks from the iTunes music store, it could be more painful still
(you’d probably decide to keep your iPod with your new Windows machine
rather than moving to a Windows music player, even though the iPod works
better with the Mac). This shows why lock-in can be so durable; although each
piece of equipment — be it a Mac laptop, an iPod, or a printer — wears out,
the lock-in persists in the complementary relationship between them. And this
doesn’t just apply to PC platforms, but to ISPs; commercial software systems
such as databases; equipment such as telephone exchanges; and various online
services.
This is why so much effort gets expended in standards wars and antitrust
suits. It’s also why so many security mechanisms now aim at controlling
compatibility. In such cases, the likely hackers are not malicious outsiders, but
the owners of the equipment, or new firms trying to challenge the incumbent
by making compatible products. The issues are made more complex by the fact
that innovation is often incremental, and products succeed when new firms
find killer applications for them [607]. The PC, for example, was designed by
IBM as a machine to run spreadsheets; if they had locked it down to this
application alone, then a massive opportunity would have been lost. Indeed,
the fact that the IBM PC was more open than the Apple Mac was a factor in its
becoming the dominant desktop platform.
So the law in many countries gives companies a right to reverse-engineer
their competitors’ products for compatibility [1110]. More and more, security
mechanisms are being used to try to circumvent that law: incumbents try to
lock down their platforms using cryptography and tamper-resistance so that
even if competitors have the legal right to try to reverse engineer them, they
are not always going to succeed in practice. Many businesses are seeing brutal
power struggles for control of the supply chain; for example, mobile phone
makers’ attempts to introduce sophisticated DRM into their handsets were frustrated by network operators determined to prevent the handset makers from

7.4 Game Theory

establishing business relationships directly with their customers. These
struggles set the scene in which more and more security products succeed
or fail.

7.3.3

Asymmetric Information

Another of the ways in which markets can fail, beyond monopoly and public
goods, is when some principals know more than others. The study of asymmetric
information was kicked off by a famous paper in 1970 on the ‘market for
lemons’ [19], for which George Akerlof won a Nobel prize. It presents the
following simple yet profound insight: suppose that there are 100 used cars
for sale in a town: 50 well-maintained cars worth $2000 each, and 50 ‘lemons’
worth $1000. The sellers know which is which, but the buyers don’t. What is
the market price of a used car? You might think $1500; but at that price no
good cars will be offered for sale. So the market price will be close to $1000.
This is one reason poor security products predominate. When users can’t tell
good from bad, they might as well buy a cheap antivirus product for $10 as a
better one for $20, and we may expect a race to the bottom on price.
A further distinction can be drawn between hidden information and hidden
action. For example, Volvo has a reputation for building safe cars that survive
accidents well, yet it is well known that Volvo drivers have more accidents.
Is this because people who know they’re bad drivers buy Volvos so they’re
less likely to get killed, or because people in Volvos drive faster? The first is
the hidden-information case, also known as adverse selection, and the second
is the hidden-action case, also known as moral hazard. Both effects are important
in security, and both may combine in specific cases. (In the case of drivers, there
seems to be a growing consensus that people adjust their driving behaviour
to keep their risk exposure to the level with which they are comfortable.
This also explains why mandatory seat-belt laws tend not to save lives
overall, merely to move fatalities from vehicle occupants to pedestrians and
cyclists [10].)
Asymmetric information explains many market failures in the real world,
from low prices in used-car markets to the difficulty that older people have
in getting insurance on reasonable terms (people who know they’re sick will
tend to buy more of it, making it uneconomic for the healthy). It tends to lead
to surveillance or rationing.

7.4

Game Theory

There are really just two ways to get something you want if you can’t find
or make it yourself. You either make something useful and trade it; or you

223

224

Chapter 7

■

Economics

take what you need, by force, by the ballot box or whatever. Choices between
cooperation and conflict are made every day at all sorts of levels, by both
humans and animals.
The main tool we can use to study and analyse them is game theory, which
I will define as ‘the study of problems of cooperation and conflict among
independent decision makers’. We’re interested in games of strategy rather
than games of chance, and we’re less interested in games of perfect information (such as chess) than in games of imperfect information, which can be
much more interesting. We try to get to the core of games by abstracting
away much of the detail. For example, consider the school playground game
of ‘matching pennies’: Alice and Bob toss coins and reveal them simultaneously, upon which Alice gets Bob’s penny if they’re different and Bob gets
Alice’s penny if they’re the same. I’ll write this as shown in Figure 7.2:

Alice

H
T

Bob
H
−1,1
1,-1

T
1,−1
−1,1

Figure 7.2: Matching pennies

Each entry in the table shows first Alice’s outcome and then Bob’s outcome.
Thus if the coins fall (H,H) Alice loses a penny and Bob gains a penny. This is
an example of a zero-sum game: Alice’s gain is Bob’s loss.
Often we can solve a game quickly by writing out a payoff matrix like this.
Here’s an example (Figure 7.3):

Alice

Bob
Left
Top
1,2
Bottom
2,1

Right
0,1
1,0

Figure 7.3: Dominant strategy equilibrium

In this game, no matter what Bob plays, Alice is better off playing ‘Bottom’;
and no matter what Alice plays, Bob is better off playing ‘Left’. Each player
has a dominant strategy — an optimal choice regardless of what the other does.
So Alice’s strategy should be a constant ‘Bottom’ and Bob’s a constant ‘Left’.
(A strategy in game theory is just an algorithm that takes a game state and
outputs a move.) We call this a dominant strategy equilibrium.

7.4 Game Theory

Another example is shown in Figure 7.4:

Alice

Bob
Left
Top
2,1
Bottom
0,0

Right
0,0
1,2

Figure 7.4: Nash equilibrium

Here each player’s optimal strategy depends on what the other player does,
or (perhaps more accurately) what they think the other player will do. We
say that two strategies are in Nash equilibrium when Alice’s choice is optimal
given Bob’s, and vice versa. Here there are two symmetric Nash equilibria, at
top left and bottom right. You can think of them as being like local optima
while a dominant strategy equilibrium is a global optimum.

7.4.1 The Prisoners’ Dilemma
We’re now ready to look at a famous problem posed in 1950 by John Nash, and
for which he won the Nobel. It applies to many situations from international
trade negotiations to free-riding in peer-to-peer file-sharing systems to cooperation between hunting animals, and Nash first studied it in the context of US
and USSR defense spending; his employer, the Rand corporation, was paid
to think about possible strategies in nuclear war. However, Nash presented it
using the following simple example.
Two prisoners are arrested on suspicion of planning a bank robbery. The
police interview them separately and tell each of them the following: ‘‘If neither
of you confesses you’ll each get a year for carrying a concealed firearm without
a permit. If one of you confesses, he’ll go free and the other will get 6 years for
conspiracy to rob. If both of you confess, you will each get three years.’’
What should the prisoners do? Let’s write the game out formally, as shown
in Figure 7.5:
Benjy
Alfie

Confess
Deny

Confess
−3,−3
−6,0

Deny
0,−6
−1,−1

Figure 7.5: The prisoners’ dilemma

When Alfie looks at this table, he will reason as follows: ‘If Benjy’s going
to confess then I should too as then I get 3 years rather than 6; and if he’s

225

226

Chapter 7

■

Economics

going to deny then I should still confess as that way I walk rather than doing
a year’. Benjy will reason similarly. The two of them will each confess, and get
three years each. This is not just a Nash equilibrium; it’s a dominant strategy
equilibrium. Each prisoner should logically confess regardless of what the
other does.
But hang on, you say, if they had agreed to keep quiet then they’ll get a
year each, which is a better outcome for them! In fact the strategy (deny,deny)
is Pareto efficient, while the dominant strategy equilibrium is not. (That’s one
reason it’s useful to have concepts like ‘Pareto efficient’ and ‘dominant strategy
equilibrium’ rather than just arguing over ‘best’.)
So what’s the solution? Well, so long as the game is going to be played once
only, and this is the only game in town, there isn’t a solution. Both prisoners
will logically confess and get three years. We can only change this state of
affairs if somehow we can change the game itself. There are many possibilities:
there can be laws of various kinds from international treaties on trade to
the gangster’s omertá. In practice, a prisoner’s dilemma game is changed by
altering the rules or the context so as to turn it into another game where the
equilibrium is more efficient.

7.4.2

Evolutionary Games

An important class of problems can be solved where the game is played
repeatedly — if Alfie and Benjy are career criminals who expect to be dealing
with each other again and again. Then of course there can be an incentive for
them to cooperate. There are at least two ways of modelling this.
In the 1970s, Bob Axelrod started thinking about how people might play
many rounds of prisoners’ dilemma. He set up a series of competitions to
which people could submit programs, and these programs played each other
repeatedly in tournaments. He found that one of the best strategies overall
was tit-for-tat, which is simply that you cooperate in round one, and at each
subsequent round you do to your opponent what he or she did in the previous
round [99]. It began to be realised that strategy evolution could explain a lot.
For example, in the presence of noise, players tend to get locked into (defect,
defect) whenever one player’s cooperative behaviour is misread by the other
as defection. So in this case it helps to ‘forgive’ the other player from time
to time.
Simultaneously, a parallel approach was opened up by John Maynard
Smith and George Price [848]. They considered what would happen if you had
a mixed population of aggressive and docile individuals, ‘hawks’ and ‘doves’,
with the behaviour that doves cooperate; hawks take food from doves; and
hawks fight, with a risk of death. Suppose the value of the food at each

7.4 Game Theory

interaction is V and the risk of death in a hawk fight per encounter is C. Then
the payoff matrix looks like Figure 7.6:

Hawk

V−C V−C
, 2
2

Hawk

Dove
V,0

Dove

0, V

V V
,
2 2

Figure 7.6: The hawk-dove game

Here, if V > C, the whole population will become hawk, as that’s the
dominant strategy, but if C > V (fighting is too expensive) then there is an
equilibrium where the probability p that a bird is a hawk sets the hawk payoff
and the dove payoff equal, that is
p

V−C
V
+ (1 − p)V = (1 − p)
2
2

which is solved by p = V/C. In other words, you can have aggressive and
docile individuals coexisting in a population, and the proportion of aggressive
individuals will at equilibrium be a function of the costs of aggression; the more
dangerous it is, the fewer such individuals there will be. Of course, the costs
can change over time, and diversity is a good thing in evolutionary terms as
a society with a minority of combative individuals may be at an advantage
when war breaks out. Again, it takes generations for a society to move to
equilibrium. Perhaps our current incidence of aggression is too high because it
reflects conditions in the Dark Ages, or even on the African highveld 500,000
years ago1 .
This neat insight, along with Bob Axelrod’s simulation methodology for
tackling problems that don’t have a neat algebraic solution, got many people
from moral philosophers to students of animal behaviour interested in evolutionary game theory. They give deep insights into how cooperation evolved.
It turns out that many primates have an inbuilt sense of fairness and punish
individuals who are seen to be cheating — the instinct for vengeance is part of
the mechanism to enforce sociality. Fairness can operate in a number of different ways at different levels. For example, the philosopher Brian Skyrms found
that doves can get a better result against hawks if they can recognise each other
and interact preferentially, giving a model for how social movements such as
freemasons and maybe even some religions establish themselves [1188].
1 A number of leading anthropologists believe that, until recent times, tribal warfare was endemic
among human societies [777].

227

228

Chapter 7

■

Economics

Of course, the basic idea behind tit-for-tat goes back a long way. The Old
Testament has ‘An eye for an eye’ and the New Testament ‘Do unto others
as you’d have them do unto you’; the latter formulation is, of course, more
fault-tolerant, and versions of it can be found in Aristotle, in Confucius and
elsewhere. More recently, Thomas Hobbes used primitive prisoners’-dilemmastyle arguments in the seventeenth century to justify the existence of a state
without the Divine Right of Kings.
The applications of evolutionary game theory keep on growing. Since 9/11,
for example, there has been interest in whether hawk-dove games explain the
ability of fundamentalists to take over discourse in religions at a time of stress.
From the economists’ viewpoint, evolutionary games explain why cartel-like
behaviour can appear in industries even where there are no secret deals being
done in smoke-filled rooms. For example, if there are three airlines operating
a profitable route, and one lowers its prices to compete for volume, the others
may well respond by cutting prices even more sharply to punish it and make
the route unprofitable, in the hope that the discounts will be discontinued and
everyone can go back to gouging the customer. And there are some interesting
applications in security, too, which I’ll come to later.

7.5

The Economics of Security and Dependability

Economists used to be well aware of the interaction between economics and
security; rich nations could afford big armies. But nowadays a web search on
‘economics’ and ‘security’ turns up relatively few articles. The main reason
is that, after 1945, economists drifted apart from people working on strategic
studies; nuclear weapons were thought to decouple national survival from
economic power [839]. A secondary factor may have been that the USA
confronted the USSR over security, but Japan and the EU over trade. It has
been left to the information security world to re-establish the connection.
One of the observations that rekindled interest in security economics came
from banking. In the USA, banks are generally liable for the costs of card
fraud; when a customer disputes a transaction, the bank must either show
she is trying to cheat it, or refund her money. In the UK, banks generally got
away with claiming that their systems were ‘secure’, and telling customers
who complained that they must be mistaken or lying. ‘Lucky bankers,’ you
might think; yet UK banks spent more on security and suffered more fraud.
This was probably a moral-hazard effect: UK bank staff knew that customer
complaints would not be taken seriously, so they became lazy and careless,
leading to an epidemic of fraud [33, 34].
Another was that people were not spending as much money on anti-virus
software as the vendors might have hoped. Now a typical virus payload then
was a service-denial attack on Microsoft; and while a rational consumer might

7.5 The Economics of Security and Dependability

spend $20 to stop a virus trashing her hard disk, she will be less likely to do so
just to protect a wealthy corporation [1290]. There are many other examples,
such as hospital systems bought by medical directors and administrators that
look after their interests but don’t protect patient privacy. The picture that
started to emerge was of system security failing because the people guarding
a system were not the people who suffered the costs of failure, or of particular
types of failure. Sometimes, as we’ll see, security mechanisms are used to
dump risks on others, and if you are one of these others you’d be better off
with an insecure system. Put differently, security is often not a scalar, but a
power relationship; the principals who control what it means in a given system
often use it to advance their own interests.
This was the initial insight. But once we started studying security economics
seriously, we found that there’s a lot more to it than that.

7.5.1 Weakest Link, or Sum of Efforts?
The late Jack Hirshleifer, the founder of conflict theory, told the story of
Anarchia, an island whose flood defences were constructed by individual
families who each maintained a section of the flood wall. The island’s flood
defence thus depended on the weakest link, that is, the laziest family. He
compared this with a city whose defences against missile attack depend on
the single best defensive shot [609]. Another example of best-shot is medieval
warfare, where there was on occasion a single combat between the two armies’
champions. Hal Varian extended this model to three cases of interest to
the dependability of information systems — where performance depends
on the minimum effort, the best effort, or the sum-of-efforts [1292]. This
last case, the sum-of-efforts, is the modern model for warfare: we pay our
taxes and the government hires soldiers. It’s a lot more efficient than best-shot
(where most people will free-ride behind the heroes), and that in turn is miles
better than weakest-link (where everyone will free-ride behind the laziest).
Program correctness can depend on minimum effort (the most careless
programmer introducing a vulnerability) while software vulnerability testing
may depend on the sum of everyone’s efforts. Security may also depend on
the best effort — the actions taken by an individual champion such as a
security architect. As more agents are added, systems become more reliable
in the total-effort case but less reliable in the weakest-link case. What are the
implications? Well, software companies should hire more software testers and
fewer (but more competent) programmers.

7.5.2 Managing the Patching Cycle
There has been much debate about ‘open source security’, and more generally
whether actively seeking and disclosing vulnerabilities is socially desirable.

229

230

Chapter 7

■

Economics

It’s a debate that has flared up again and again; as we saw in the preface, the
Victorians agonised over whether it was socially responsible to publish books
about lockpicking, and eventually concluded that it was [1257]. People have
worried more recently about the online availability of (for example) the US
Army Improvised Munitions Handbook [1271]; is the risk of helping terrorists
sufficient to justify online censorship?
Security economics provides both a theoretical and a quantitative framework
for discussing some issues of this kind. I showed in 2002 that, under standard
assumptions of reliability growth, open systems and proprietary systems are
just as secure as each other; opening up a system helps the attackers and defenders equally [54]. Thus the open-security question will often be an empirical one,
turning on the extent to which a given real system follows the standard model.
In 2004, Eric Rescorla argued that for software with many latent vulnerabilities, removing one bug makes little difference to the likelihood of an attacker
finding another one later. Since exploits are often based on vulnerabilities
inferred from patches, he argued against disclosure and frequent patching
unless the same vulnerabilities are likely to be rediscovered [1071]. Ashish
Arora and others responded with data showing that public disclosure made
vendors respond with fixes more quickly; attacks increased to begin with,
but reported vulnerabilities declined over time [88]. In 2006, Andy Ozment
and Stuart Schechter found that the rate at which unique vulnerabilities were
disclosed for the core OpenBSD operating system has decreased over a six-year
period [998]. These results support the current system of responsible disclosure whereby people who discover vulnerabilities report them to CERT, which
reports them on to vendors, and publicises them once patches are shipped.
This is by no means all that there is to say about the economics of dependability. There are tensions between vendors and their customers over the
frequency and timing of patch release; issues with complementers; difficulties
with metrics; companies such as iDefense and TippingPoint that buy and sell
information on vulnerabilities; and even concerns that intelligence agencies
with privileged access to bug reports use them for zero-day exploits against
other countries’ systems. I’ll come back to all this in Part III.

7.5.3

Why Is Windows So Insecure?

The micromanagement of the patching cycle begs a deeper question: why
are there so many bugs in the first place? In particular, why is Windows
so insecure, despite Microsoft’s dominant market position? It’s possible to
write much better software, and there are fields such as defense and healthcare
where a serious effort is made to produce dependable systems. Why do
we not see a comparable effort made with commodity platforms, especially
since Microsoft has no real competitors?

7.5 The Economics of Security and Dependability

To be honest, Microsoft’s software security is improving. Windows 95
was dreadful, Windows 98 slightly better, and the improvement’s continued
through NT, XP and Vista. But the attackers are getting better too, and the
protection in Vista isn’t all for the user’s benefit. As Peter Gutmann points
out, enormous effort has gone into protecting premium video content, and
almost no effort into protecting users’ credit card numbers [570]. The same
pattern has also been seen in other platform products, from the old IBM
mainframe operating systems through telephone exchange switches to the
Symbian operating system for mobile phones. Products are insecure at first,
and although they improve over time, many of the new security features are
for the vendor’s benefit as much as the user’s.
By now, you should not find this surprising. The combination of high fixed
and low marginal costs, network effects and technical lock-in makes platform
markets particularly likely to be dominated by single vendors, who stand to
gain vast fortunes if they can win the race to dominate the market. In such
a race, the notorious Microsoft philosophy of the 1990s — ‘ship it Tuesday
and get it right by version 3’ — is perfectly rational behaviour. In such a race,
the platform vendor must appeal at least as much to complementers — to the
software companies who decide whether to write applications for its platform
or for someone else’s. Security gets in the way of applications, and it tends
to be a lemons market anyway. So the rational vendor will enable (indeed
encourage) all applications to run as root, until his position is secure. Then he
will add more security — but there will still be a strong incentive to engineer
it in such a way as to maximise customer lock-in, or to appeal to complementers
in new markets such as digital media.
From the viewpoint of the consumer, markets with lock-in are often ‘bargains
then rip-offs’. You buy a nice new printer for $39.95, then find to your disgust
after just a few months that you need two new printer cartridges for $19.95
each. You wonder whether you’d not be better off just buying a new printer.
From the viewpoint of the application developer, markets with standards races
based on lock-in look a bit like this. At first it’s really easy to write code for
them; later on, once you’re committed, there are many more hoops to jump
through. From the viewpoint of the poor consumer, they could be described
as ‘poor security, then security for someone else’.
Sometimes it can be worse than that. When racing to establish a dominant
position, vendors are likely to engineer their products so that the cost of
managing such security as there is falls on the user, rather than on the
application developers. A classic example is SSL/TLS encryption. This was
adopted in the mid-1990s as Microsoft and Netscape battled for dominance of
the browser market. As I discussed in Chapter 2, SSL leaves it up to the user
to assess the certificate offered by a web site and decide whether to trust it;
and this has turned out to facilitate all kinds of phishing and other attacks. Yet
dumping the compliance costs on the user made perfect sense at the time, and

231

232

Chapter 7

■

Economics

competing protocols such as SET, that would have required heavier investment
by banks and merchants, were allowed to wither on the vine [357]. The world
has ended up not just with a quite insecure system of Internet payments,
but with widespread liability dumping that makes progress towards a better
system difficult. Too much of the infrastructure has weakest-link rather than
sum-of-efforts security.

7.5.4

Economics of Privacy

The big conundrum with privacy is that people say that they value privacy,
yet act otherwise. If you stop people in the street and ask them their views,
about a third say they are privacy fundamentalists and will never hand over
their personal information to marketers or anyone else; about a third say they
don’t care; and about a third are in the middle, saying they’d take a pragmatic
view of the risks and benefits of any disclosure. However, the behavior that
people exhibit via their shopping behavior — both online and offline — is
quite different; the great majority of people pay little heed to privacy, and will
give away the most sensitive information for little benefit. Privacy-enhancing
technologies have been offered for sale by various firms, yet most have failed
in the marketplace. Why should this be?
Privacy is one aspect of information security that interested economists
before 2000. In 1978, Richard Posner defined privacy in terms of secrecy [1035],
and the following year extended this to seclusion [1036]. In 1980, Jack
Hirshleifer published a seminal paper in which he argued that rather than
being about withdrawing from society, privacy was a means of organising
society, arising from evolved territorial behavior; internalised respect for property is what allows autonomy to persist in society. These privacy debates in
the 1970s led in Europe to generic data-protection laws, while the USA limited
itself to a few sector-specific laws such as HIPAA. Economists’ appetite for
work on privacy was further whetted recently by the Internet, the dotcom
boom, and the exploding trade in personal information about online shoppers.
An early modern view of privacy can be found in a 1996 paper by Hal Varian
who analysed privacy in terms of information markets [1289]. Consumers
want to not be annoyed by irrelevant marketing calls while marketers do
not want to waste effort. Yet both are frustrated, because of search costs,
externalities and other factors. Varian suggested giving consumers rights in
information about themselves, and letting them lease it to marketers with the
proviso that it not be resold without permission.
The recent proliferation of complex, information-intensive business models
demanded a broader approach. Andrew Odlyzko argued in 2003 that privacy
erosion is a consequence of the desire to charge different prices for similar
services [981]. Technology is simultaneously increasing both the incentives
and the opportunities for price discrimination. Companies can mine online

7.5 The Economics of Security and Dependability

purchases and interactions for data revealing individuals’ willingness to
pay. From airline yield-management systems to complex and ever-changing
software and telecommunications prices, differential pricing is economically
efficient — but increasingly resented. Peter Swire argued that we should
measure the costs of privacy intrusion more broadly [1237]. If a telesales
operator calls 100 prospects, sells three of them insurance, and annoys 80,
then the conventional analysis considers only the benefit to the three and to
the insurer. However, persistent annoyance causes millions of people to go
ex-directory, to not answer the phone during dinner, or to screen calls through
an answering machine. The long-run societal harm can be considerable.
Several empirical studies have backed this up by examining people’s privacy
valuations.
My own view on this is that it simply takes time for the public to assimilate
the privacy risks. For thirty years or so, IT policy folks have been agonising
about the death of privacy, but this remained a geek interest until recently.
The significance is now starting to percolate down to sophisticated people
like stock-market investors: Alessandro Acquisti and others have found that
the stock price of companies reporting a security or privacy breach is likely
to fall [8, 265]. It’s only when tabloid newspapers and talk-radio shows give
lots of coverage to stories of ordinary people who’ve suffered real harm as a
result of ‘identity theft’ and phishing that the average voter will start to sit
up and take notice. There are some early signs that this is starting to happen
(for example in the growing number of requests that privacy experts like me
get to appear on radio and TV shows). But another behavioural economist,
George Loewnstein, points out that people are more sensitive to large changes
in their circumstances rather than to small ones: they will get concerned if
things suddenly get worse, but not if they get worse gradually. They also
become habituated surprisingly easily to bad circumstances that they don’t
believe they can change.
It may be of particular interest that, in late 2007, the British government
suffered spectacular embarrassment when it lost the electronic tax records
on all the nation’s children and their families — including bank account
details — leading to a personal apology in Parliament from the Prime Minister
and massive media coverage of subsequent privacy breaches. I’ll discuss this
in more detail in section 9.4; the privacy economist’s interest will be in whether
this changes public attitudes in any measurable way over time, and whether
attitudes stay changed.

7.5.5 Economics of DRM
Rights-management technologies have also come in for economic scrutiny.
Hal Varian pointed out in 2002 that DRM and similar mechanisms were also
about tying, bundling and price discrimination; and that their unfettered use

233

234

Chapter 7

■

Economics

could damage competition [1291]. I wrote an FAQ on ‘Trusted Computing’
in 2003, followed by a research paper, in which I pointed out the potential
for competitive abuse of rights management mechanisms; for example, by
transferring control of user data from the owner of the machine on which it is
stored to the creator of the file in which it is stored, the potential for lock-in
is hugely increased [53]. Think of the example above, in which a law firm
of 100 fee earners each has a PC on which they install Office for $500. The
$50,000 they pay Microsoft is roughly equal to the total costs of switching to
(say) OpenOffice, including training, converting files and so on. However, if
control of the files moves to its thousands of customers, and the firm now
has to contact each customer and request a digital certificate in order to
migrate the file, then clearly the switching costs have increased — so you can
expect the cost of Office to increase too, over time.
There are some interesting angles on the debate about rights management in
music too. In 2004, Felix Oberholzer and Koleman Strumpf published a nowfamous paper, in which they examined how music downloads and record
sales were correlated [978]. They showed that downloads do not do significant
harm to the music industry. Even in the most pessimistic interpretation,
five thousand downloads are needed to displace a single album sale, while
high-selling albums actually benefit from file sharing. This research was
hotly disputed by music-industry spokesmen at the time, but has since been
confirmed by Canadian government research that found a positive correlation
between downloading and CD sales among peer-to-peer system users, and no
correlation among the population as a whole [28].
In January 2005, Hal Varian made a controversial prediction [1293]: that
stronger DRM would help system vendors more than the music industry,
because the computer industry is more concentrated (with only three serious suppliers of DRM platforms — Microsoft, Sony, and the dominant firm,
Apple). The content industry scoffed, but by the end of that year music publishers were protesting that Apple was getting too large a share of the cash
from online music sales. As power in the supply chain moved from the music
majors to the platform vendors, so power in the music industry appears to
be shifting from the majors to the independents, just as airline deregulation
favoured aircraft makers and low-cost airlines. This is a striking demonstration
of the predictive power of economic analysis. By fighting a non-existent threat,
the record industry helped the computer industry forge a weapon that may be
its undoing.

7.6

Summary

Many systems fail because the incentives are wrong, rather than because of
some technical design mistake. As a result, the security engineer needs to

Further Reading

understand basic economics as well as the basics of crypto, protocols, access
controls and psychology. Security economics is a rapidly growing research area
that explains many of the things that we used to consider just ‘bad weather’,
such as the insecurity of Windows. It constantly throws up fascinating new
insights into all sorts of questions from how to optimise the patching cycle
through whether people really care about privacy to what legislators might do
about DRM.

Research Problems
So far, two areas of economics have been explored for their relevance to
security, namely microeconomics and game theory. Behavioural economics
(the boundary between economics and psychology) has also started to yield
insights. But economics is a vast subject. What other ideas might it give us?

Further Reading
The best initial introduction to information economics is Shapiro and Varian’s ‘Information Rules’ which remains remarkably fresh and accurate for a
book written ten years ago [1159]. I generally recommend that students read
this first. For those who want to go on to do research in the subject, I then
suggest Hal Varian’s textbook ‘Intermediate Microeconomics’ which covers the
material from an academic viewpoint, with fewer case histories and more
mathematics [1284].
The current research in security economics is published mostly at the Workshop on the Economics of Information Security (WEIS), which has been held
annually since 2002; details of WEIS, and other relevant events, can be found
at [58]. There is a current (2007) survey of the field, that I wrote with Tyler
Moore, at [72]. There are two books of collected research papers [257, 548], and
a popular account by Bruce Schneier [1129]; I also maintain an Economics and
Security Resource Page at http://www.cl.cam.ac.uk/ rja14/econsec.html.
Two other relevant papers, which are in press as this book goes to print,
are a report I’m writing with Rainer Böhme, Richard Clayton and Tyler
Moore on security economics in the European internal market [62], and an
OECD report by Michel van Eeten and colleagues that reports extensive interviews with information security stakeholders about the incentives they face in
practice [420].
A number of economists study related areas. I mentioned Jack Hirshleifer’s
conflict theory; a number of his papers are available in a book [610]. Another
really important strand is the economics of crime, which was kick-started
by Gary Becker [138], and has recently been popularised by Steve Levitt and

235

236

Chapter 7

■

Economics

Stephen Dubner’s ‘Freakonomics’ [787]. Much of this analyses volume crime
by deprived young males, an issue to which I’ll return in Chapter 11; but some
scholars have also studied organised crime [392, 473]. As computer crime is
increasingly driven by the profit motive rather than by ego and bragging
rights, we can expect economic analyses to be ever more useful.

PART

II
In this second part of the book, I describe a large
number of applications of secure systems, many of
which introduce particular protection concepts or
technologies.
There are three broad themes. Chapters 8–10 look
at conventional computer security issues, and by discussing what one is trying to do and how it’s done
in different environments — the military, banks and
healthcare — we introduce security policy models
which set out the protection concepts that real systems
try to implement. We introduce our first detailed case
studies. An example is the worldwide network of automatic teller machines, which illustrates many of the
problems of transferring familiar protection properties
from a bank branch to a global distributed environment
using cryptography.
Chapters 10–18 look at the hardware and system
engineering aspects of information security. This includes biometrics, the design of various tokens such
as smartcards, and the whole panoply of hardware
security mechanisms from physical locks through chiplevel tamper-resistance and emission security to security printing and seals. New applications that illustrate
the technologies are described, ranging from electronic warfare and nuclear weapon control through
burglar alarms, truck speed limiters and prepayment
gas meters. We end up with a chapter on the security of
application programming interfaces, where hardware
and software security meet.

238

Part II

The third theme is attacks on networks, and on highly-networked systems. We start off with electronic and information warfare in Chapter 19, as
these activities give some of the more extreme examples and show how far
techniques of denial, deception and exploitation can be taken by a resourceful
opponent under severe operational pressure. It also gives a view of surveillance
and intrusion from the intelligence agencies’ point of view, and introduces
concepts such as anonymity and traffic analysis. We then study the lessons
of history by examining frauds on phone systems, and on applications that
rely on them, in Chapter 20. This sets the scene for a discussion in Chapter 21
of attacks on computer networks and defensive technologies ranging from
firewalls to protocols such as SSH, TLS, IPSEC and Bluetooth. Chapter 22
deals with copyright and DRM, and related topics such as information hiding.
Finally, in Chapter 23 I present four vignettes of the ‘bleeding edge’ of security
research in 2007: computer games; web applications (such as auctions, search
and social networking); privacy technology (from email anonymity and web
proxies to forensics countermeasures); and elections.
One reason for this ordering is to give the chapters a logical progression.
Thus, for example, I discuss frauds against magnetic stripe bank cards before
going on to describe the smartcards which may replace them and the pay-TV
systems which actually use smartcards today.
Sometimes a neat linear ordering isn’t possible as a particular technology has
evolved through a number of iterations over several applications. In such cases
I try to distill what I know into a case history. To keep the book manageable
for readers who use it primarily as a reference rather than as a textbook, I have
put the more technical material towards the end of each chapter or section: if
you get lost at a first reading then do just skip to the next section and carry on.

CHAPTER

8
Multilevel Security
Most high assurance work has been done in the area of kinetic devices
and infernal machines that are controlled by stupid robots. As
information processing technology becomes more important
to society, these concerns spread to areas
previously thought inherently harmless,
like operating systems.
— Earl Boebert
I brief;
you leak;
he/she commits a criminal offence
by divulging classified information.
— British Civil Service Verb
They constantly try to escape
From the darkness outside and within
By dreaming of systems so perfect that no one will need to be good
— TS Eliot

8.1

Introduction

I mentioned in the introduction that military database systems, which can
hold information at a number of different levels of classification (Confidential,
Secret, Top Secret, . . .), have to ensure that data can only be read by a principal
whose level is at least as high as the data’s classification. The policies they
implement are known as multilevel secure or alternatively as mandatory access
control or MAC.

239

240

Chapter 8

■

Multilevel Security

Multilevel secure systems are important because:
1. a huge amount of research has been done on them, thanks to military
funding for computer science in the USA. So the military model of protection has been worked out in much more detail than any other, and it gives
us a lot of examples of the second-order and even third-order effects of
implementing a security policy rigorously;
2. although multilevel concepts were originally developed to support confidentiality in military systems, many commercial systems now use multilevel integrity policies. For example, telecomms operators want their
billing system to be able to see what’s happening in their switching system, but not affect it;
3. recently, products such as Microsoft Vista and Red Hat Linux have started
to incorporate mandatory access control mechanisms, and they have also
appeared in disguise in digital rights management systems. For example,
Red Hat uses SELinux mechanisms developed by the NSA to isolate different servers running on a machine — so that even if your web server is
hacked, it doesn’t necessarily follow that your DNS server gets taken over
too. Vista has a multilevel integrity policy under which Internet Explorer
runs by default at ‘Low’ — which means that even if it gets taken over,
the attacker should not be able to change system files, or anything else
with a higher integrity level. These mechanisms are still largely invisible
to the domestic computer user, but their professional use is increasing;
4. multilevel confidentiality ideas are often applied in environments where
they’re ineffective or even harmful, because of the huge vested interests and momentum behind them. This can be a contributory factor in
the failure of large system projects, especially in the public sector.
Sir Isiah Berlin famously described thinkers as either foxes or hedgehogs:
a fox knows many little things, while a hedgehog knows one big thing. The
multilevel philosophy is the hedgehog approach to security engineering.

8.2

What Is a Security Policy Model?

Where a top-down approach to security engineering is possible, it will typically take the form of threat model — security policy — security mechanisms. The
critical, and often neglected, part of this process is the security policy.
By a security policy, we mean a document that expresses clearly and
concisely what the protection mechanisms are to achieve. It is driven by our
understanding of threats, and in turn drives our system design. It will often
take the form of statements about which users may access which data. It
plays the same role in specifying the system’s protection requirements, and

8.2 What Is a Security Policy Model?

evaluating whether they have been met, that the system specification does
for general functionality. Indeed, a security policy may be part of a system
specification, and like the specification its primary function is to communicate.
Many organizations use the phrase ‘security policy’ to mean a collection of
vapid statements. Figure 8.1 gives a simple example:
Megacorp Inc security policy
1. This policy is approved by Management.
2. All staff shall obey this security policy.
3. Data shall be available only to those with a ‘need-to-know’.
4. All breaches of this policy shall be reported at once to Security.
Figure 8.1: A typical corporate information security policy

This sort of waffle is very common but is useless to the security engineer.
Its first failing is it dodges the central issue, namely ‘Who determines ‘‘needto-know’’ and how?’ Second, it mixes statements at a number of different
levels (organizational approval of a policy should logically not be part of the
policy itself). Third, there is a mechanism but it’s implied rather than explicit:
‘staff shall obey’ — but what does this mean they actually have to do? Must the
obedience be enforced by the system, or are users ‘on their honour’? Fourth,
how are breaches to be detected and who has a specific duty to report them?
We must do better than this. In fact, because the term ‘security policy’ is
widely abused to mean a collection of managerialist platitudes, there are three
more precise terms which have come into use to describe the specification of
protection requirements.
A security policy model is a succinct statement of the protection properties
which a system, or generic type of system, must have. Its key points can
typically be written down in a page or less. It is the document in which
the protection goals of the system are agreed with an entire community, or
with the top management of a customer. It may also be the basis of formal
mathematical analysis.
A security target is a more detailed description of the protection mechanisms
that a specific implementation provides, and how they relate to a list of control
objectives (some but not all of which are typically derived from the policy
model). The security target forms the basis for testing and evaluation of a
product.
A protection profile is like a security target but expressed in an implementation-independent way to enable comparable evaluations across products and
versions. This can involve the use of a semi-formal language, or at least of
suitable security jargon. A protection profile is a requirement for products that
are to be evaluated under the Common Criteria [935]. (I discuss the Common

241

242

Chapter 8

■

Multilevel Security

Criteria in Part III; they are associated with a scheme used by many governments for mutual recognition of security evaluations of defense information
systems.)
When I don’t have to be so precise, I may use the phrase ‘security policy’ to
refer to either a security policy model or a security target. I will never use it
to refer to a collection of platitudes.
Sometimes, we are confronted with a completely new application and have
to design a security policy model from scratch. More commonly, there already
exists a model; we just have to choose the right one, and develop it into a
security target. Neither of these steps is easy. Indeed one of the purposes of this
section of the book is to provide a number of security policy models, describe
them in the context of real systems, and examine the engineering mechanisms
(and associated constraints) which a security target can use to meet them.
Finally, you may come across a third usage of the phrase ‘security policy’ — as a list of specific configuration settings for some protection product.
We will refer to this as configuration management, or occasionally as trusted
configuration management, in what follows.

8.3

The Bell-LaPadula Security Policy Model

The classic example of a security policy model was proposed by Bell and
LaPadula in 1973, in response to US Air Force concerns over the security of
time-sharing mainframe systems1 . By the early 1970’s, people had realised that
the protection offered by many commercial operating systems was poor, and
was not getting any better. As soon as one operating system bug was fixed,
some other vulnerability would be discovered. (Modern reliability growth
models can quantify this and confirm that the pessimism was justified; I
discuss them further in section 26.2.4.) There was the constant worry that even
unskilled users would discover loopholes and use them opportunistically;
there was also a keen and growing awareness of the threat from malicious
code. (Viruses were not invented until the following decade; the 70’s concern
was about Trojans.) There was a serious scare when it was discovered that
the Pentagon’s World Wide Military Command and Control System was
vulnerable to Trojan Horse attacks; this had the effect of restricting its use
to people with a ‘Top Secret’ clearance, which was inconvenient. Finally,
academic and industrial researchers were coming up with some interesting
new ideas on protection, which I discuss below.
A study by James Anderson led the US government to conclude that a
secure system should do one or two things well; and that these protection
1
This built on the work of a number of other researchers: see section 9.2.1 below for a sketch of
the technical history.

8.3 The Bell-LaPadula Security Policy Model

properties should be enforced by mechanisms which were simple enough to
verify and that would change only rarely [29]. It introduced the concept of a
reference monitor — a component of the operating system which would mediate
access control decisions and be small enough to be subject to analysis and
tests, the completeness of which could be assured. In modern parlance, such
components — together with their associated operating procedures — make
up the Trusted Computing Base (TCB). More formally, the TCB is defined as the
set of components (hardware, software, human, . . .) whose correct functioning
is sufficient to ensure that the security policy is enforced, or, more vividly,
whose failure could cause a breach of the security policy. The Anderson
report’s goal was to make the security policy simple enough for the TCB to be
amenable to careful verification.
But what are these core security properties that should be enforced above
all others?

8.3.1 Classifications and Clearances
The Second World War, and the Cold War which followed, led NATO
governments to move to a common protective marking scheme for labelling
the sensitivity of documents. Classifications are labels, which run upwards from
Unclassified through Confidential, Secret and Top Secret (see Figure 8.2.). The
details change from time to time. The original idea was that information
whose compromise could cost lives was marked ‘Secret’ while information
whose compromise could cost many lives was ‘Top Secret’. Government
employees have clearances depending on the care with which they’ve been
vetted; in the USA, for example, a ‘Secret’ clearance involves checking FBI
fingerprint files, while ‘Top Secret’ also involves background checks for the
previous five to fifteen years’ employment [379].
The access control policy was simple: an official could read a document
only if his clearance was at least as high as the document’s classification. So
an official cleared to ‘Top Secret’ could read a ‘Secret’ document, but not vice
versa. The effect is that information may only flow upwards, from confidential
to secret to top secret, but it may never flow downwards unless an authorized
person takes a deliberate decision to declassify it.
There are also document handling rules; thus a ‘Confidential’ document
might be kept in a locked filing cabinet in an ordinary government office,
TOP SECRET
SECRET
CONFIDENTIAL
UNCLASSIFIED
Figure 8.2: Multilevel security

243

244

Chapter 8

■

Multilevel Security

while higher levels may require safes of an approved type, guarded rooms
with control over photocopiers, and so on. (The NSA security manual [952]
gives a summary of the procedures used with ‘top secret’ intelligence data.)
The system rapidly became more complicated. The damage criteria for
classifying documents were expanded from possible military consequences
to economic harm and even political embarrassment. The UK has an extra
level, ‘Restricted’, between ‘Unclassified’ and ‘Confidential’; the USA used
to have this too but abolished it after the Freedom of Information Act was
introduced. America now has two more specific markings: ‘For Official Use
only’ (FOUO) refers to unclassified data that can’t be released under FOIA,
while ‘Unclassified but Sensitive’ includes FOUO plus material which might be
released in response to a FOIA request. In the UK, ‘Restricted’ information is in
practice shared freely, but marking everything ‘Restricted’ allows journalists
and others involved in leaks to be prosecuted under Official Secrets law.
(Its other main practical effect is that an unclassified US document which
is sent across the Atlantic automatically becomes ‘Restricted’ in the UK and
then ‘Confidential’ when shipped back to the USA. American military system
builders complain that the UK policy breaks the US classification scheme!)
There is also a system of codewords whereby information, especially at
Secret and above, can be further restricted. For example, information which
might reveal intelligence sources or methods — such as the identities of agents
or decrypts of foreign government traffic — is typically classified ‘Top Secret
Special Compartmented Intelligence’ or TS/SCI, which means that so-called
need to know restrictions are imposed as well, with one or more codewords
attached to a file. Some of the codewords relate to a particular military
operation or intelligence source and are available only to a group of named
users. To read a document, a user must have all the codewords that are
attached to it. A classification label, plus a set of codewords, makes up a
security category or (if there’s at least one codeword) a compartment, which is
a set of records with the same access control policy. I discuss compartmentation
in more detail in the chapter on multilateral security.
There are also descriptors, caveats and IDO markings. Descriptors are words
such as ‘Management’, ‘Budget’, and ‘Appointments’: they do not invoke any
special handling requirements, so we can deal with a file marked ‘Confidential — Management’ as if it were simply marked ‘Confidential’. Caveats are
warnings such as ‘UK Eyes Only’, or the US equivalent, ‘NOFORN’; there are
also International Defence Organisation markings such as NATO. The lack of
obvious differences between codewords, descriptors, caveats and IDO marking is one of the things that can make the system confusing. A more detailed
explanation can be found in [1051].
The final generic comment about access control doctrine is that allowing
upward-only flow of information also models what happens in wiretapping.
In the old days, tapping someone’s telephone meant adding a physical wire

8.3 The Bell-LaPadula Security Policy Model

at the exchange; nowadays, it’s all done in the telephone exchange software
and the effect is somewhat like making the target calls into conference calls
with an extra participant. The usual security requirement is that the target
of investigation should not know he is being wiretapped, so the third party
should be silent — and its very presence must remain unknown to the target.
For example, now that wiretaps are implemented as silent conference calls,
care has to be taken to ensure that the charge for the conference call facility goes
to the wiretapper, not to the target. Wiretapping requires an information flow
policy in which the ‘High’ principal can see ‘Low’ data, but a ‘Low’ principal
can’t tell whether ‘High’ is reading any data at all, let alone what data.

8.3.2 Information Flow Control
It was in this context of the classification of government data that the BellLaPadula or BLP model of computer security was formulated in 1973 [146]. It
is also known as multilevel security and systems which implement it are often
called multilevel secure or MLS systems. Their basic property is that information
cannot flow downwards.
More formally, the Bell-LaPadula model enforces two properties:
The simple security property: no process may read data at a higher level.
This is also known as no read up (NRU);
The *-property: no process may write data to a lower level. This is also
known as no write down (NWD).
The *-property was Bell and LaPadula’s critical innovation. It was driven
by the fear of attacks using malicious code. An uncleared user might write a
Trojan and leave it around where a system administrator cleared to ‘Secret’
might execute it; it could then copy itself into the ‘Secret’ part of the system,
read the data there and try to signal it down somehow. It’s also quite possible
that an enemy agent could get a job at a commercial software house and embed
some code in a product which would look for secret documents to copy. If it
could then write them down to where its creator could read it, the security
policy would have been violated. Information might also be leaked as a result
of a bug, if applications could write down.
Vulnerabilities such as malicious and buggy code are assumed to be given. It
is also assumed that most staff are careless, and some are dishonest; extensive
operational security measures have long been used, especially in defence
environments, to prevent people leaking paper documents. (When I worked
in defense avionics as a youngster, all copies of circuit diagrams, blueprints
etc were numbered and had to be accounted for.) So there was a pre-existing
culture that security policy was enforced independently of user actions; the
move to computers didn’t change this. It had to be clarified, which is what BellLaPadula does: the security policy must be enforced not just independently of

245

246

Chapter 8

■

Multilevel Security

users’ direct actions, but of their indirect actions (such as the actions taken by
programs they run).
So we must prevent programs running at ‘Secret’ from writing to files at
‘Unclassified’, or more generally prevent any process at High from signalling
to any object (or subject) at Low. In general, when systems enforce a security
policy independently of user actions, they are described as having mandatory
access control, as opposed to the discretionary access control in systems like Unix
where users can take their own access decisions about their files.
The Bell-LaPadula model makes it relatively straightforward to verify claims
about the protection provided by a design. Given both the simple security
property (no read up), and the star property (no write down), various results
can be proved about the machine states which can be reached from a given
starting state, and this simplifies formal analysis. There are some elaborations,
such as a trusted subject — a principal who is allowed to declassify files. To keep
things simple, we’ll ignore this; we’ll also ignore the possibility of incompatible
security levels for the time being, and return to them in the next chapter; and
finally, in order to simplify matters still further, we will assume from now
on that the system has only two levels, High and Low (unless there is some
particular reason to name individual compartments).
Multilevel security can be implemented in a number of ways. The original
idea was to implement a reference monitor by beefing up the part of an
operating system which supervises all operating system calls and checks
access permissions to decide whether the call can be serviced or not. However
in practice things get much more complex as it’s often hard to build systems
whose trusted computing base is substantially less than the whole operating
system kernel (plus quite a number of its utilities).
Another approach that has been gaining ground as hardware has got cheaper
and faster is to replicate systems. This replication was often physical in the
1990s, and since about 2005 it may use virtual machines; some promising
recent work builds on virtualization products such as VMware and Xen to
provide multiple systems at different security levels on the same PC. One
might, for example, have one database running at Low and another at High,
on separate instances of Windows XP, with a pump that constantly copies
information from Low up to High, all running on VMware on top of SELinux.
I’ll discuss pumps in more detail later.

8.3.3

The Standard Criticisms of Bell-LaPadula

The introduction of BLP caused a lot of excitement: here was a straightforward
security policy which was clear to the intuitive understanding yet still allowed
people to prove theorems. But John McLean showed that the BLP rules
were not in themselves enough. He introduced System Z, defined as a BLP
system with the added feature that a user can ask the system administrator to

8.3 The Bell-LaPadula Security Policy Model

temporarily declassify any file from High to Low. In this way, Low users can
read any High file without breaking the BLP assumptions.
Bell’s argument was that System Z cheats by doing something the model
doesn’t allow (changing labels isn’t a valid operation on the state), and
McLean’s argument was that it didn’t explicitly tell him so. The issue is dealt
with by introducing a tranquility property. The strong tranquility property says
that security labels never change during system operation, while the weak
tranquility property says that labels never change in such a way as to violate
a defined security policy.
The motivation for the weak property is that in a real system we often
want to observe the principle of least privilege and start off a process at the
uncleared level, even if the owner of the process were cleared to ‘Top Secret’.
If she then accesses a confidential email, her session is automatically upgraded
to ‘Confidential’; and in general, her process is upgraded each time it accesses
data at a higher level (this is known as the high water mark principle). As
subjects are usually an abstraction of the memory management sub-system
and file handles, rather than processes, this means that state changes when
access rights change, rather than when data actually moves.
The practical implication is that a process acquires the security labels of all
the files it reads, and these become the default label set of every file that it writes.
So a process which has read files at ‘Secret’ and ‘Crypto’ will thereafter create
files marked (at least) ‘Secret Crypto’. This will include temporary copies made
of other files. If it then reads a file at ‘Top Secret Daffodil’ then all files it creates
after that will be labelled ‘Top Secret Crypto Daffodil’, and it will not be able to
write to any temporary files at ‘Secret Crypto’. The effect this has on applications is one of the serious complexities of multilevel security; most application
software needs to be rewritten (or at least modified) to run on MLS platforms.
Read-time changes in security level introduce the problem that access to
resources can be revoked at any time, including in the middle of a transaction.
Now the revocation problem is generally unsolvable in modern operating
systems, at least in any complete form, which means that the applications
have to cope somehow. Unless you invest some care and effort, you can easily
find that everything ends up in the highest compartment — or that the system
fragments into thousands of tiny compartments that don’t communicate at all
with each other. I’ll discuss this in more detail in the next chapter.
Another problem with BLP, and indeed with all mandatory access control
systems, is that separating users and processes is relatively straightforward; the
hard part is when some controlled interaction is needed. Most real applications
need some kind of ‘trusted subject’ that can break the security policy; an
example is a trusted word processor that helps an intelligence analyst scrub a
Top Secret document when she’s editing it down to Secret [861]. BLP is silent
on how the system should protect such an application. Does it become part of
the Trusted Computing Base? I’ll discuss this in more detail below.

247

248

Chapter 8

■

Multilevel Security

Finally it’s worth noting that even with the high-water-mark refinement,
BLP still doesn’t deal with the creation or destruction of subjects or objects
(which is one of the hard problems of building a real MLS system).

8.3.4

Alternative Formulations

Multilevel security properties have been expressed in several other ways.
The first multilevel security policy was a version of high water mark
written in 1967–8 for the ADEPT-50, a mandatory access control system
developed for the IBM S/360 mainframe [1334]. This used triples of level,
compartment and group, with the groups being files, users, terminals and jobs.
As programs (rather than processes) were subjects, it was vulnerable to Trojan
horse compromises, and it was more complex than need be. Nonetheless, it
laid the foundation for BLP, and also led to the current IBM S/390 mainframe
hardware security architecture [632].
Shortly after that, a number of teams produced primitive versions of the
lattice model, which I’ll discuss in more detail in the next chapter. These also
made a significant contribution to the Bell-LaPadula work, as did engineers
working on Multics. Multics had started as an MIT project in 1965 and
developed into a Honeywell product; it became the template for the ‘trusted
systems’ specified in the Orange Book, being the inspirational example of the
B2 level operating system. The evaluation that was carried out on it by Paul
Karger and Roger Schell was hugely influential and was the first appearance of
the idea that malware could be hidden in the compiler [693] — which led to Ken
Thompson’s famous paper ‘On Trusting Trust’ ten years later. Multics itself
developed into a system called SCOMP that I’ll discuss in section 8.4.1 below.
Noninterference was introduced by Joseph Goguen and Jose Meseguer in
1982 [532]. In a system with this property, High’s actions have no effect on
what Low can see. Nondeducibility is less restrictive and was introduced by
David Sutherland in 1986 [1233]. Here the idea is to try and prove that Low
cannot deduce anything with 100 percent certainty about High’s input.
Low users can see High actions, just not understand them; a more formal
definition is that any legal string of high level inputs is compatible with every
string of low level events. So for every trace Low can see, there’s a similar
trace that didn’t involve High input. But different low-level event streams may
require changes to high-level outputs or reordering of high-level/low-level
event sequences.
The motive for nondeducibility is to find a model that can deal with
applications such as a LAN on which there are machines at both Low and
High, with the High machines encrypting their LAN traffic. (Quite a lot else is
needed to do this right, from padding the High traffic with nulls so that Low
users can’t do traffic analysis, and even ensuring that the packets are the same
size — see [1096] for an early example of such a system.)

8.3 The Bell-LaPadula Security Policy Model

Nondeducibility has historical importance since it was the first nondeterministic version of Goguen and Messeguer’s ideas. But it is hopelessly weak.
There’s nothing to stop Low making deductions about High input with 99%
certainty. There’s also a whole lot of problems when we are trying to prove
results about databases, and have to take into account any information which
can be inferred from data structures (such as from partial views of data with
redundancy) as well as considering the traces of executing programs. I’ll
discuss these problems further in the next chapter.
Improved models include Generalized Noninterference and restrictiveness. The
former is the requirement that if one alters a high level input event in a legal
sequence of system events, the resulting sequence can be made legal by, at
most, altering one or more subsequent high-level output events. The latter
adds a further restriction on the part of the trace where the alteration of the
high-level outputs can take place. This is needed for technical reasons to ensure
that two systems satisfying the restrictiveness property can be composed into
a third which also does. See [864] which explains these issues.
The Harrison-Ruzzo-Ullman model tackles the problem of how to deal with
the creation and deletion of files, an issue on which BLP is silent. It operates on
access matrices and verifies whether there is a sequence of instructions which
causes an access right to leak to somewhere it was initially not present [584].
This is more expressive than BLP, but is more complex and thus less tractable
as an aid to verification.
John Woodward proposed a Compartmented Mode Workstation (CMW) policy,
which attempted to model the classification of information using floating
labels, as opposed to the fixed labels associated with BLP [1357, 552]. It was
ultimately unsuccessful, because labels tend to either float up too far too
fast (if done correctly), or they float up more slowly (but don’t block all the
opportunities for malicious information flow). However, CMW ideas have
led to real products — albeit products that provide separation more than
information sharing.
The type enforcement model, due to Earl Boebert and Dick Kain [198], assigns
subjects to domains and objects to types, with matrices defining permitted
domain-domain and domain-type interactions. This is used in a popular and
important mandatory access control system, SELinux, which simplifies it by
putting both subjects and objects in types and having a matrix of allowed
type pairs [813]. In effect this is a second access-control matrix; in addition
to having a user ID and group ID, each process has a security ID. The Linux
Security Modules framework provides pluggable security modules with rules
operating on SIDs.
Type enforcement was later extended by Badger and others to Domain and
Type Enforcement [106]. They introduced their own language for configuration
(DTEL), and implicit typing of files based on pathname; for example, all
objects in a given subdirectory may be declared to be in a given domain. TE

249

250

Chapter 8

■

Multilevel Security

and DTE are more general than simple MLS policies such as BLP, as they
start to deal with integrity as well as confidentiality concerns. One of their
early uses, starting in the LOCK system, was to enforce trusted pipelines: the
idea is to confine a set of trusted processes in a pipeline so that each can
only talk to previous stage and the next stage. This can be used to assemble
guards and firewalls which cannot be bypassed unless at least two stages are
compromised [963]. Type-enforcement mechanisms are used, for example, in
the Sidewinder firewall. A further advantage of type enforcement mechanisms
is that they can be aware of code versus data, and privileges can be bound to
code; in consequence the tranquility problem can be dealt with at execute time
rather than as data are read. This can make things much more tractable.
The downside of the greater flexibility and expressiveness of TE/DTE is
that it is not always straightforward to implement BLP, because of the state
explosion; when writing a security policy you have to consider all the possible
interactions between different types. (For this reason, SELinux also implements
a simple MLS policy. I’ll discuss SELinux in more detail below.)
Finally, a policy model getting much attention from researchers in recent
years is role-based access control (RBAC), introduced by David Ferraiolo and
Richard Kuhn [466, 467]. This provides a more general framework for mandatory access control than BLP in which access decisions don’t depend on users’
names but on the functions which they are currently performing within the
organization. Transactions which may be performed by holders of a given role
are specified, then mechanisms for granting membership of a role (including
delegation). Roles, or groups, had for years been the mechanism used in
practice in organizations such as banks to manage access control; the RBAC
model starts to formalize this. It can be used to give finer-grained control,
for example by granting different access rights to ‘Ross as Professor’, ‘Ross as
member of the Planning and Resources Committee’ and ‘Ross reading private
email’. Implementations vary; the banking systems of twenty years ago kept
the controls in middleware, and some modern RBAC products do control
at the application layer where it’s easy to bypass. SELinux builds it on top of
TE, so that users are mapped to roles at login time, roles are authorized for
domains and domains are given permissions to types. On such a platform,
RBAC can usefully deal with integrity issues as well as confidentiality, by
allowing role membership to be revised when certain programs are invoked.
Thus, for example, a process calling untrusted software that had been downloaded from the net might lose the role membership required to write to
sensitive system files.

8.3.5

The Biba Model and Vista

The incorporation into Windows Vista of a multilevel integrity model has
revived interest in a security model devised in 1975 by Ken Biba [168], which

8.3 The Bell-LaPadula Security Policy Model

textbooks often refer to as ‘Bell-LaPadula upside down’. The Biba model deals
with integrity alone and ignores confidentiality. The key observation is that
confidentiality and integrity are in some sense dual concepts — confidentiality
is a constraint on who can read a message, while integrity is a constraint on
who can write or alter it.
As a concrete application, an electronic medical device such as an ECG
may have two separate modes: calibration and use. Calibration data must be
protected from corruption by normal users, who will therefore be able to read
it but not write to it; when a normal user resets the device, it will lose its
current user state (i.e., any patient data in memory) but the calibration will
remain unchanged.
To model such a system, we can use a multilevel integrity policy with the
rules that we can read data at higher levels (i.e., a user process can read
the calibration data) and write to lower levels (i.e., a calibration process can
write to a buffer in a user process); but we must never read down or write
up, as either could allow High integrity objects to become contaminated
with Low — that is potentially unreliable — data. The Biba model is often
formulated in terms of the low water mark principle, which is the dual of the high
water mark principle discussed above: the integrity of an object is the lowest
level of all the objects that contributed to its creation.
This was the first formal model of integrity. A surprisingly large number of
real systems work along Biba lines. For example, the passenger information
system in a railroad may get information from the signalling system, but
certainly shouldn’t be able to affect it (other than through a trusted interface,
such as one of the control staff). However, few of the people who build such
systems are aware of the Biba model or what it might teach them.
Vista marks file objects with an integrity level, which can be Low, Medium,
High or System, and implements a default policy of NoWriteUp. Critical Vista
files are at System and other objects are at Medium by default — except for
Internet Explorer which is at Low. The effect is that things downloaded using
IE can read most files in a Vista system, but cannot write them. The idea is to
limit the damage that can be done by viruses and other malware. I’ll describe
Vista’s mechanisms in more detail below.
An interesting precursor to Vista was LOMAC, a Linux extension that
implemented a low water mark policy [494]. It provided two levels — high
and low integrity — with system files at High and the network at Low. As
soon as a program (such as a daemon) received traffic from the network, it
was automatically downgraded to Low. Thus even if the traffic contains an
attack that forks a root shell, this shell could not write to the password file as a
normal root shell would. As one might expect, a number of system tasks (such
as logging) became tricky and required trusted code.
As you might expect, Biba has the same fundamental problems as BellLaPadula. It cannot accommodate real-world operation very well without

251

252

Chapter 8

■

Multilevel Security

numerous exceptions. For example, a real system will usually require ‘trusted’
subjects that can override the security model, but Biba on its own fails to
provide effective mechanisms to protect and confine them; and in general it
doesn’t work so well with modern software environments. In the end, Vista
dropped the NoReadDown restriction and did not end up using its integrity
model to protect the base system from users.
Biba also cannot express many real integrity goals, like assured pipelines.
In fact, the Type Enforcement model was introduced by Boebert and Kain as
an alternative to Biba. It is unfortunate that Vista didn’t incorporate TE.
I will consider more complex models when I discuss banking and bookkeeping systems in Chapter 10; these are more complex in that they retain
security state in the form of dual control mechanisms, audit trails and so on.

8.4

Historical Examples of MLS Systems

Following some research products in the late 1970’s (such as KSOS [166],
a kernelised secure version of Unix), products that implemented multilevel
security policies started arriving in dribs and drabs in the early 1980’s. By
about 1988, a number of companies started implementing MLS versions of
their operating systems. MLS concepts were extended to all sorts of products.

8.4.1

SCOMP

One of the most important products was the secure communications processor (SCOMP), a derivative of Multics launched in 1983 [491]. This was a
no-expense-spared implementation of what the US Department of Defense
believed it wanted for handling messaging at multiple levels of classification.
It had formally verified hardware and software, with a minimal kernel and
four rings of protection (rather than Multics’ seven) to keep things simple. Its
operating system, STOP, used these rings to maintain up to 32 separate compartments, and to allow appropriate one-way information flows between them.
SCOMP was used in applications such as military mail guards. These are
specialised firewalls which typically allow mail to pass from Low to High
but not vice versa [369]. (In general, a device which does this is known as
a data diode.) SCOMP’s successor, XTS-300, supported C2G, the Command
and Control Guard. This was used in the time phased force deployment
data (TPFDD) system whose function was to plan US troop movements
and associated logistics. Military plans are developed as TPFDDs at a high
classification level, and then distributed at the appropriate times as commands
to lower levels for implementation. (The issue of how high information is
deliberately downgraded raises a number of issues, some of which I’ll deal

8.4 Historical Examples of MLS Systems

with below. In the case of TPFDD, the guard examines the content of each
record before deciding whether to release it.)
SCOMP’s most significant contribution was to serve as a model for the
Orange Book [375] — the US Trusted Computer Systems Evaluation Criteria.
This was the first systematic set of standards for secure computer systems,
being introduced in 1985 and finally retired in December 2000. The Orange
Book was enormously influential not just in the USA but among allied powers;
countries such as the UK, Germany, and Canada based their own national
standards on it, until these national standards were finally subsumed into the
Common Criteria [935].
The Orange Book allowed systems to be evaluated at a number of levels
with A1 being the highest, and moving downwards through B3, B2, B1 and
C2 to C1. SCOMP was the first system to be rated A1. It was also extensively
documented in the open literature. Being first, and being fairly public, it set
the standard for the next generation of military systems. This standard has
rarely been met since; in fact, the XTS-300 was only evaluated to B3 (the formal
proofs of correctness required for an A1 evaluation were dropped).

8.4.2 Blacker
Blacker was a series of encryption devices designed to incorporate MLS
technology. Previously, encryption devices were built with separate processors
for the ciphertext, or Black, end and the cleartext, or Red, end. Various possible
failures can be prevented if one can coordinate the Red and Black processing.
One can also make the device simpler, and provide greater operational
flexibility: the device isn’t limited to separating two logical networks, but can
provide encryption and integrity assurance selectively, and interact in useful
ways with routers. But then a high level of assurance is required that the ‘Red’
data won’t leak out via the ‘Black’.
Blacker entered service in 1989, and the main lesson learned from it was
the extreme difficulty of accommodating administrative traffic within a model
of classification levels [1335]. As late as 1994, it was the only communications
security device with an A1 evaluation [161]. So it too had an effect on later
systems. It was not widely used though, and its successor (the Motorola
Network Encryption System), had only a B2 evaluation.

8.4.3 MLS Unix and Compartmented Mode Workstations
MLS versions of Unix started to appear in the late 1980’s, such as AT&T’s System V/MLS [27]. This added security levels and labels, initially by using some
of the bits in the group id record and later by using this to point to a more elaborate structure. This enabled MLS properties to be introduced with minimal
changes to the system kernel. Other products of this kind included SecureWare

253

254

Chapter 8

■

Multilevel Security

(and its derivatives, such as SCO and HP VirtualVault), and Addamax. By the
time of writing (2007), Sun’s Solaris has emerged as the clear market leader,
being the platform of choice for high-assurance server systems and for many
clients as well. Trusted Solaris 8 gave way to Solaris trusted Extensions 10,
which has now been folded into Solaris, so that every copy of Solaris contains
MLS mechanisms, for those knowledgeable enough to use them.
Comparted Mode Workstations (CMWs) are an example of MLS clients. They
allow data at different levels to be viewed and modified at the same time
by a human operator, and ensure that labels attached to the information
are updated appropriately. The initial demand came from the intelligence
community, whose analysts may have access to ‘Top Secret’ data, such as
decrypts and agent reports, and produce reports at the ‘Secret’ level for users
such as political leaders and officers in the field. As these reports are vulnerable
to capture, they must not contain any information which would compromise
intelligence sources and methods.
CMWs allow an analyst to view the ‘Top Secret’ data in one window,
compose a report in another, and have mechanisms to prevent the accidental
copying of the former into the latter (i.e., cut-and-paste works from ‘Secret’ to
‘Top Secret’ but not vice versa). CMWs have proved useful in operations, logistics and drug enforcement as well [631]. For the engineering issues involved in
doing mandatory access control in windowing systems, see [437, 438] which
describe a prototype for Trusted X, a system implementing MLS but not information labelling. It runs one instance of X Windows per sensitivity level, and
has a small amount of trusted code which allows users to cut and paste from
a lower level to a higher one. For the specific architectural issues with Sun’s
CMW product, see [451].

8.4.4

The NRL Pump

It was soon realised that simple mail guards and crypto boxes were too
restrictive, as many more networked services were developed besides mail.
Traditional MLS mechanisms (such as blind write-ups and periodic readdowns) are inefficient for real-time services.
The US Naval Research Laboratory (NRL) therefore developed the Pump — a
one-way data transfer device (a data diode) to allow secure one-way information flow (Figure 8.3). The main problem is that while sending data from
Low to High is easy, the need for assured transmission reliability means
that acknowledgement messages must be sent back from High to Low. The
Pump limits the bandwidth of possible backward leakage using a number of
mechanisms such as using buffering and randomizing the timing of acknowledgements [685, 687, 688]. The attraction of this approach is that one can build
MLS systems by using pumps to connect separate systems at different security
levels. As these systems don’t process data at more than one level, they can be

8.4 Historical Examples of MLS Systems

HIGH

PUMP

LOW
Figure 8.3: The NRL pump

built from cheap commercial-off-the-shelf (COTS) components [689]. As the
cost of hardware falls, this is often the preferred option where it’s possible.
The pump’s story is told in [691].
The Australian government developed a product called Starlight that uses
pump-type technology married with a keyboard switch to provide a nice
MLS-type windowing system (albeit without any visible labels) using a bit
of trusted hardware which connects the keyboard and mouse with High and
Low systems [30]. There is no trusted software. It’s been integrated with the
NRL Pump [689]. A number of semi-commercial data diode products have
also been introduced.

8.4.5

Logistics Systems

Military stores, like government documents, can have different classification
levels. Some signals intelligence equipment is ‘Top Secret’, while things
like jet fuel and bootlaces are not; but even such simple commodities may
become ‘Secret’ when their quantities or movements might leak information
about tactical intentions. There are also some peculiarities: for example, an
inertial navigation system classified ‘Confidential’ in the peacetime inventory
might contain a laser gyro platform classified ‘Secret’ (thus security levels are
nonmonotonic).
The systems needed to manage all this seem to be hard to build, as MLS
logistics projects in both the USA and UK have ended up as expensive
disasters. In the UK, the Royal Air Force’s Logistics Information Technology
System (LITS) was a 10 year (1989–99), £500m project to provide a single
stores management system for the RAF’s 80 bases [932]. It was designed to
operate on two levels: ‘Restricted’ for the jet fuel and boot polish, and ‘Secret’
for special stores such as nuclear bombs. It was initially implemented as two

255

256

Chapter 8

■

Multilevel Security

separate database systems connected by a pump to enforce the MLS property.
The project became a classic tale of escalating costs driven by creeping
requirements changes. One of these changes was the easing of classification
rules with the end of the Cold War. As a result, it was found that almost
all the ‘Secret’ information was now static (e.g., operating manuals for airdrop nuclear bombs which are now kept in strategic stockpiles rather than at
airbases). In order to save money, the ‘Secret’ information is now kept on a CD
and locked up in a safe.
Logistics systems often have application security features too. The classic example is that ordnance control systems alert users who are about to
breach safety rules by putting explosives and detonators in the same truck or
magazine [910].

8.4.6

Sybard Suite

Most governments’ information security agencies have been unable to resist
user demands to run standard applications (such as MS Office) which are
not available for multilevel secure platforms. One response was the ‘Purple
Penelope’ software, from Qinetiq in the UK, now sold as Sybard Suite. This
puts an MLS wrapper round a Windows workstation, implementing the high
water mark version of BLP. It displays in the background the current security
level of the device and upgrades it when necessary as more sensitive resources
are read. It ensures that the resulting work product is labelled correctly.
Rather than preventing users from downgrading, as a classical BLP system
might do, it allows them to assign any security label they like to their output.
However, if this involves a downgrade, the user must confirm the release of
the data using a trusted path interface, thus ensuring no Trojan Horse or virus
can release anything completely unnoticed. Of course, a really clever malicious
program can piggy-back classified material on stuff that the user does wish to
release, so there are other tricks to make that harder. There is also an audit
trail to provide a record of all downgrades, so that errors and attacks (whether
by users, or by malware) can be traced after the fact [1032]. The security policy
was described to me by one of its authors as ‘we accept that we can’t stop
people leaking the order of battle to the Guardian newspaper if they really
want to; we just want to make sure we arrest the right person for it.’

8.4.7

Wiretap Systems

One of the large applications of MLS is in wiretapping systems. Communications intelligence is generally fragile; once a target knows his traffic is
being read he can usually do something to frustrate it. Traditional wiretap
kit, based on ‘loop extenders’ spliced into the line, could often be detected by
competent targets; modern digital systems try to avoid these problems, and

8.5 Future MLS Systems

provide a multilevel model in which multiple agencies at different levels can
monitor a target, and each other; the police might be tapping a drug dealer,
and an anti-corruption unit watching the police, and so on. Wiretaps are
commonly implemented as conference calls with a silent third party, and the
main protection goal is to eliminate any covert channels that might disclose
the existence of surveillance. This is not always met. For a survey, see [1161],
which also points out that the pure MLS security policy is insufficient: suspects
can confuse wiretapping equipment by introducing bogus signalling tones.
The policy should thus have included resistance against online tampering.
Another secondary protection goal should have been to protect against
software tampering. In a recent notorious case, a wiretap was discovered on
the mobile phones of the Greek Prime Minister and his senior colleagues; this
involved unauthorised software in the mobile phone company’s switchgear
that abused the lawful intercept facility. It was detected when the buggers’
modifications caused some text messages not to be delivered [1042]. The
phone company was fined 76 million Euros (almost $100m). Perhaps phone
companies will be less willing to report unauthorized wiretaps in future.

8.5

Future MLS Systems

In the first edition of this book, I wrote that the MLS industry’s attempts to
market its products as platforms for firewalls, web servers and other exposed
systems were failing because ‘the BLP controls do not provide enough of
a protection benefit in many commercial environments to justify their large
development costs, and widely fielded products are often better because of
the evolution that results from large-scale user feedback’. I also noted research
on using mandatory access controls to accommodate both confidentiality and
integrity in environments such as smartcards [692], and to provide real-time
performance guarantees to prevent service denial attacks [889]. I ventured that
‘perhaps the real future of multilevel systems is not in confidentiality, but
integrity’.
The last seven years appear to have proved this right.

8.5.1

Vista

Multilevel integrity is coming to the mass market in Vista. As I already
mentioned, Vista essentially uses the Biba model. All processes do, and all
securable objects (including directories, files and registry keys) may, have an
integrity-level label. File objects are labelled at ‘Medium’ by default, while
Internet Explorer (and everything downloaded using it) is labelled ‘Low’. User
action is therefore needed to upgrade downloaded content before it can modify

257

258

Chapter 8

■

Multilevel Security

existing files. This may not be a panacea: it may become so routine a demand
from all installed software that users will be trained to meekly upgrade viruses
too on request. And it must be borne in mind that much of the spyware infesting
the average home PC was installed there deliberately (albeit carelessly and with
incomplete knowledge of the consequences) after visiting some commercial
website. This overlap between desired and undesired software sets a limit on
how much can be achieved against downloaded malware. We will have to
wait and see.
It is also possible to implement a crude BLP policy using Vista, as you can
also set ‘NoReadUp’ and ‘NoExecuteUp’ policies. These are not installed as
default; the reason appears to be that Microsoft was principally concerned
about malware installing itself in the system and then hiding. Keeping the
browser ‘Low’ makes installation harder, and allowing all processes (even
Low ones) to inspect the rest of the system makes hiding harder. But it does
mean that malware running at Low can steal all your data; so some users
might care to set ‘NoReadUp’ for sensitive directories. No doubt this will
break a number of applications, so a cautious user might care to have separate
accounts for web browsing, email and sensitive projects. This is all discussed
by Joanna Rutkowska in [1099]; she also describes some interesting potential
attacks based on virtualization. A further problem is that Vista, in protected
mode, does still write to high-integrity parts of the registry, even though
Microsoft says it shouldn’t [555].
In passing, it’s also worth mentioning rights management, whether of the
classical DRM kind or the more recent IRM (Information Rights Management)
variety, as a case of mandatory access control. Vista, for example, tries to
ensure that no high definition video content is ever available to an untrusted
process. I will discuss it in more detail later, but for now I’ll just remark that
many of the things that go wrong with multilevel systems might also become
vulnerabilities in, or impediments to the use of, rights-management systems.
Conversely, the efforts expended by opponents of rights management in trying
to hack the Vista DRM mechanisms may also open up holes in its integrity
protection.

8.5.2

Linux

The case of SELinux and Red Hat is somewhat similar to Vista in that the
immediate goal of the new mandatory access control mechanisms is also
to limit the effects of a compromise. SELinux [813] is based on the Flask
security architecture [1209], which separates the policy from the enforcement
mechanism; a security context contains all of the security attributes associated
with a subject or object in Flask, where one of those attributes includes
the Type Enforcement type attribute. A security identifier is a handle to a
security context, mapped by the security server. It has a security server where

8.5 Future MLS Systems

policy decisions are made, this resides in-kernel since Linux has a monolithic
kernel and the designers did not want to require a kernel-userspace call
for security decisions (especially as some occur on critical paths where the
kernel is holding locks) [557]). The server provides a general security API to
the rest of the kernel, with the security model hidden behind that API. The
server internally implements RBAC, TE, and MLS (or to be precise, a general
constraints engine that can express MLS or any other model you like). SELinux
is included in a number of Linux distributions, and Red Hat’s use is typical.
There its function is to separate various services. Thus an attacker who takes
over your web server does not thereby acquire your DNS server as well.
Suse Linux has taken a different path to the same goal. It uses AppArmor,
a monitoring mechanism maintained by Novell, which keeps a list of all the
paths each protected application uses and prevents it accessing any new ones.
It is claimed to be easier to use than the SELinux model; but operating-system
experts distrust it as it relies on pathnames as the basis for its decisions. In
consequence, it has ambiguous and mutable identifiers; no system view of
subjects and objects; no uniform abstraction for handling non-file objects; and
no useful information for runtime files (such as /tmp). By forcing policy to
be written in terms of individual objects and filesystem layout rather than
security equivalence classes, it makes policy harder to analyze. However, in
practice, with either AppArmor or SELinux, you instrument the code you plan
to protect, watch for some months what it does, and work out a policy that
allows it to do just what it needs. Even so, after you have fielded it, you will
still have to observe and act on bug reports for a year or so. Modern software
components tend to be so complex that figuring out what access they need is
an empirical and iterative process2 .
It’s also worth bearing in mind that simple integrity controls merely stop
malware taking over the machine — they don’t stop it infecting a Low compartment and using that as a springboard from which to spread elsewhere.
Integrity protection is not the only use of SELinux. At present there is
considerable excitement about it in some sections of government, who are
excited at the prospect of ‘cross department access to data . . . and a common
trust infrastructure for shared services’ (UK Cabinet Office) and allowing ‘users
to access multiple independent sessions at varying classification levels’ (US
Coast Guard) [505]. Replacing multiple terminals with single ones, and moving
from proprietary systems to open ones, is attractive for many reasons — and
providing simple separation between multiple terminal emulators or browsers
running on the same PC is straightforward. However, traditional MAC might
not be the only way to do it.
2
Indeed, some of the mandatory access control mechanisms promised in Vista — such as remote
attestation — did not ship in the first version, and there have been many papers from folks
at Microsoft Research on ways of managing access control and security policy in complex
middleware. Draw your own conclusions.

259

260

Chapter 8

8.5.3

■

Multilevel Security

Virtualization

Another technological approach is virtualization. Products such as VMware
and Xen are being used to provide multiple virtual machines at different
levels. Indeed, the NSA has produced a hardened version of VMware, called
NetTop, which is optimised for running several Windows virtual machines
on top of an SELinux platform. This holds out the prospect of giving the
users what they want — computers that have the look and feel of ordinary
windows boxes — while simultaneously giving the security folks what they
want, namely high-assurance separation between material at different levels
of classification. So far, there is little information available on NetTop, but it
appears to do separation rather than sharing.
A current limit is the sheer technical complexity of modern PCs; it’s very
difficult to find out what things like graphics cards actually do, and thus to
get high assurance that they don’t contain huge covert channels. It can also
be quite difficult to ensure that a device such as a microphone or camera is
really connected to the Secret virtual machine rather than the Unclassified
one. However, given the effort being put by Microsoft into assurance for
high-definition video content, there’s at least the prospect that some COTS
machines might eventually offer reasonable assurance on I/O eventually.
The next question must be whether mandatory access control for confidentiality, as opposed to integrity, will make its way out of the government sector
and into the corporate world. The simplest application might be for a company
to provide its employees with separate virtual laptops for corporate and home
use (whether with virtualisation or with mandatory access controls). From the
engineering viewpoint, virtualization might preferable, as it’s not clear that
corporate security managers will want much information flow between the
two virtual laptops: flows from ‘home’ to ‘work’ could introduce malware
while flow from ‘work’ to ‘home’ could leak corporate secrets. From the business viewpoint, it’s less clear that virtualization will take off. Many corporates
would rather pretend that company laptops don’t get used for anything else,
and as the software industry generally charges per virtual machine rather than
per machine, there could be nontrivial costs involved. I expect most companies
will continue to ignore the problem and just fire people whose machines cause
visible trouble.
The hardest problem is often managing the interfaces between levels, as
people usually end up having to get material from one level to another in
order to get their work done. If the information flows are limited and easy
to model, as with the Pump and the CMW, well and good; the way forward
may well be Pump or CMW functionality also hosted on virtual machines.
(Virtualisation per se doesn’t give you CMW — you need a trusted client for
that, or an app running on a trusted server — but it’s possible to envisage a
trusted thin client plus VMs at two different levels all running on the same box.

8.6 What Goes Wrong

So virtualization should probably be seen as complementary to mandatory
access control, rather than a competitor.
But many things can go wrong, as I will discuss in the next session.

8.5.4 Embedded Systems
There are more and more fielded systems which implement some variant of the
Biba model. As well as the medical-device and railroad signalling applications
already mentioned, there are utilities. In an electricity utility, for example,
operational systems such as power dispatching should not be affected by any
others. The metering systems can be observed by, but not influenced by, the
billing system. Both billing and power dispatching feed information into fraud
detection, and at the end of the chain the executive information systems can
observe everything while having no direct effect on operations. These one-way
information flows can be implemented using mandatory access controls and
there are signs that, given growing concerns about the vulnerability of critical
infrastructure, some utilities are starting to look at SELinux.
There are many military embedded systems too. The primitive mail guards
of 20 years ago have by now been supplanted by guards that pass not just
email but chat, web services and streaming media, often based on SELinux;
an example is described in [478]. There are many more esoteric applications:
for example, some US radars won’t display the velocity of a US aircraft whose
performance is classified, unless the operator has the appropriate clearance.
(This has always struck me as overkill, as he can just use a stopwatch.)
Anyway, it’s now clear that many of the lessons learned in the early
multilevel systems go across to a number of applications of much wider
interest. So do a number of the failure modes, which I’ll now discuss.

8.6

What Goes Wrong

As I’ve frequently pointed out, engineers learn more from the systems that fail
than from those that succeed, and MLS systems have certainly been an effective
teacher. The large effort expended in building systems to follow a simple policy
with a high level of assurance has led to the elucidation of many second- and
third-order consequences of information flow controls. I’ll start with the more
theoretical and work through to the business and engineering end.

8.6.1 Composability
Consider a simple device that accepts two ‘High’ inputs H1 and H2 ; multiplexes them; encrypts them by xor’ing them with a one-time pad (i.e., a
random generator); outputs the other copy of the pad on H3 ; and outputs the

261

262

Chapter 8

■

Multilevel Security

RAND

H3

•



H2



H1



XOR

L

XOR

Figure 8.4: Insecure composition of secure systems with feedback

ciphertext, which being encrypted with a cipher system giving perfect secrecy,
is considered to be low (output L), as in Figure 8.4.
In isolation, this device is provably secure. However, if feedback is permitted,
then the output from H3 can be fed back into H2 , with the result that the high
input H1 now appears at the low output L. Timing inconsistencies can also
lead to the composition of two secure systems being insecure (see for example
McCullough [854]). Simple information flow doesn’t compose; neither does
noninterference or nondeducibility.
In general, the problem of how to compose two or more secure components
into a secure system is hard, even at the relatively uncluttered level of proving
results about ideal components. Most of the low-level problems arise when
some sort of feedback is introduced into the system; without it, composition
can be achieved under a number of formal models [865]. However, in real
life, feedback is pervasive, and composition of security properties can be
made even harder by detailed interface issues, feature interactions and so on.
For example, one system might produce data at such a rate as to perform a
service-denial attack on another. (I’ll discuss some of the interface problems
with reference monitors in detail in Chapter 18, ‘API Attacks’.)
Finally, the composition of secure components or systems is very often
frustrated by higher-level incompatibilities. Components might have been
designed in accordance with two different security policies, or designed
according to requirements that are inconsistent or even incompatible. This is
bad enough for different variants on the BLP theme but even worse when one
of the policies is of a non-BLP type, as we will encounter in the following two
chapters. Composability is a long-standing and very serious problem with
trustworthy systems; a good recent survey is the final report of the CHATS
project [963].

8.6.2

The Cascade Problem

An example of the difficulty of composing multilevel secure systems is given
by the cascade problem (Figure 8.5). After the Orange book introduced a series

8.6 What Goes Wrong

Top Secret

Secret

Secret

Unclassified

Figure 8.5: The cascade problem

of graduated evaluation levels, this led to rules about the number of levels
which a system can span [379]. For example, a system evaluated to B3 was
in general allowed to process information at Unclassified, Confidential and
Secret, or at Confidential, Secret and Top Secret; there was no system permitted
to process Unclassified and Top Secret data simultaneously [379].
As the diagram shows, it is straightforward to connect together two B3
systems in such a way that this security policy is broken. The first system
connects together Unclassified, Confidential and Secret, and its Confidential
and Secret levels communicate with the second system which also processes
Top Secret information. (The problem’s discussed in more detail in [622].)
This illustrates another kind of danger which formal models of security (and
practical implementations) must take into account.

8.6.3 Covert Channels
One of the reasons why these span limits are imposed on multilevel systems
emerges from a famous — and extensively studied — problem: the covert channel. First pointed out by Lampson in 1973 [768], a covert channel is a mechanism
that was not designed for communication but which can nonetheless be abused
to allow information to be communicated down from High to Low.

263

264

Chapter 8

■

Multilevel Security

A typical covert channel arises when a high process can signal to a low
one by affecting some shared resource. For example, it could position the
disk head at the outside of the drive at time ti to signal that the i-th bit in
a Top Secret file was a 1, and position it at the inside to signal that the bit
was a 0.
All systems with shared resources must find a balance between covert
channel capacity, resource utilization, and fairness. If a machine is shared
between high and low, and resources are not allocated in fixed slices, then the
high process can signal by filling up the disk drive, or by using a lot of CPU or
bus cycles (some people call the former case a storage channel and the latter a
timing channel, though in practice they can often be converted into each other).
There are many others such as sequential process IDs, shared file locks and last
access times on files — reimplementing all of these in a multilevel secure way
is an enormous task. Various strategies have been adopted to minimize their
bandwidth; for example, we can arrange that the scheduler assigns a fixed
disk quota to each level, and reads the boot sector each time control is passed
downwards; and we might also allocate a fixed proportion of the available time
slices to processes at each level, and change these proportions infrequently.
Each change might allow one or more bits to be signalled, but such strategies
can enormously reduce the available bandwidth. (A more complex multilevel
design, which uses local schedulers at each level plus a global scheduler to
maintain overall consistency, is described in [686].)
It is also possible to limit the covert channel capacity by introducing noise.
Some machines have had randomised system clocks for this purpose. But
some covert channel capacity almost always remains. (Techniques to analyze
the trade-offs between covert channel capacity and system performance are
discussed in [554].)
Many covert channels occur at the application layer, and are a real concern
to security engineers (especially as they are often overlooked). An example
from social care is a UK proposal to create a national database of all children,
for child-protection and welfare purposes, containing a list of all professionals
with which each child has contact. Now it may be innocuous that child X
is registered with family doctor Y, but the fact of a child’s registration with
a social work department is not innocuous at all — it’s well known to be
stigmatizing. For example, teachers will have lower expectations of children
whom they know to have been in contact with social workers. So it is quite
reasonable for parents (and children) to want to keep any record of such
contact private [66].
A more subtle example is that in general personal health information derived
from visits to genitourinary medicine clinics is High in the sense that it can’t
be shared with the patient’s normal doctor and thus appear in their normal
medical record (Low) unless the patient consents. In one case, a woman’s visit
to a GUM clinic leaked when the insurer failed to recall her for a smear test

8.6 What Goes Wrong

which her normal doctor knew was due [886]. The insurer knew that a smear
test had been done already by the clinic, and didn’t want to pay twice.
Another case of general interest arises in multilevel integrity systems such
as banking and utility billing, where a programmer who has inserted Trojan
code in a bookkeeping system can turn off the billing to an account by a
certain pattern of behavior (in a phone system he might call three numbers in
succession, for example). Code review is the only real way to block such attacks,
though balancing controls can also help in the specific case of bookkeeping.
The highest-bandwidth covert channel of which I’m aware is also a feature
of a specific application. It occurs in large early warning radar systems,
where High — the radar processor — controls hundreds of antenna elements
that illuminate Low — the target — with high speed pulse trains that are
modulated with pseudorandom noise to make jamming harder. In this case,
the radar code must be trusted as the covert channel bandwidth is many
megabits per second.
The best that developers have been able to do consistently with BLP
confidentiality protection in regular time-sharing operating systems is to limit
it to 1 bit per second or so. (That is a DoD target [376], and techniques for doing
a systematic analysis may be found in Kemmerer [706].) One bit per second
may be tolerable in an environment where we wish to prevent large TS/SCI
files — such as satellite photographs — leaking down from TS/SCI users to
‘Secret’ users. It is much less than the rate at which malicious code might hide
data in outgoing traffic that would be approved by a guard. However, it is
inadequate if we want to prevent the leakage of a cryptographic key. This is
one of the reasons for the military doctrine of doing crypto in special purpose
hardware rather than in software.

8.6.4 The Threat from Viruses
The vast majority of viruses are found on mass-market products such as
PCs. However, the defense computer community was shocked when Cohen
used viruses to penetrate multilevel secure systems easily in 1983. In his first
experiment, a file virus which took only eight hours to write managed to
penetrate a system previously believed to be multilevel secure [311].
There are a number of ways in which viruses and other malicious code
can be used to perform such attacks. If the reference monitor (or other TCB
components) can be corrupted, then a virus could deliver the entire system to
the attacker, for example by issuing him with an unauthorised clearance. For
this reason, slightly looser rules apply to so-called closed security environments
which are defined to be those where ‘system applications are adequately
protected against the insertion of malicious logic’ [379]. But even if the TCB
remains intact, the virus could still use any available covert channel to signal
information down.

265

266

Chapter 8

■

Multilevel Security

So in many cases a TCB will provide some protection against viral attacks,
as well as against careless disclosure by users or application software — which
is often more important than malicious disclosure. However, the main effect
of viruses on military doctrine has been to strengthen the perceived case for
multilevel security. The argument goes that even if personnel can be trusted,
one cannot rely on technical measures short of total isolation to prevent viruses
moving up the system, so one must do whatever reasonably possible to stop
them signalling back down.

8.6.5

Polyinstantiation

Another problem that has much exercised the research community is polyinstantiation. Suppose that our High user has created a file named agents, and
our Low user now tries to do the same. If the MLS operating system prohibits
him, it will have leaked information — namely that there is a file called agents
at High. But if it lets him, it will now have two files with the same name.
Often we can solve the problem by a naming convention, which could be
as simple as giving Low and High users different directories. But the problem
remains a hard one for databases [1112]. Suppose that a High user allocates
a classified cargo to a ship. The system will not divulge this information to a
Low user, who might think the ship is empty, and try to allocate it another
cargo or even to change its destination.
The solution favoured in the USA for such systems is that the High user
allocates a Low cover story at the same time as the real High cargo. Thus the
underlying data will look something like Figure 8.6.
In the UK, the theory is simpler — the system will automatically reply
‘classified’ to a Low user who tries to see or alter a High record. The two
available views would be as in Figure 8.7.
Level

Cargo

Destination

Secret
Restricted
Unclassified

Missiles
—
Engine spares

Iran
—
Cyprus

Figure 8.6: How the USA deals with classified data

Level

Cargo

Destination

Secret
Restricted
Unclassified

Missiles
Classified
—

Iran
Classified
—

Figure 8.7: How the UK deals with classified data

8.6 What Goes Wrong

This makes the system engineering simpler. It also prevents the mistakes
and covert channels which can still arise with cover stories (e.g., a Low user
tries to add a container of ammunition for Cyprus). The drawback is that
everyone tends to need the highest available clearance in order to get their
work done. (In practice, of course, cover stories still get used in order not to
advertise the existence of a covert mission any more than need be.)
There may be an interesting new application to the world of online gaming.
Different countries have different rules about online content; for example,
the USA limits online gambling, while Germany has strict prohibitions on the
display of swastikas and other insignia of the Third Reich. Now suppose a
second-world-war reenactment society wants to operate in Second Life. If
a German resident sees flags with swastikas, an offence is committed there.
Linden Labs, the operator of Second Life, has suggested authenticating users’
jurisdictions; but it’s not enough just to exclude Germans, as one of them
might look over the fence. An alternative proposal is to tag alternative objects
for visibility, so that a German looking at the Battle of Kursk would see only
inoffensive symbols. Similarly, an American looking at an online casino might
just see a church instead. Here too the lie has its limits; when the American
tries to visit that church he’ll find that he can’t get through the door.

8.6.6 Other Practical Problems
Multilevel secure systems are surprisingly expensive and difficult to build and
deploy. There are many sources of cost and confusion.
1. MLS systems are built in small volumes, and often to high standards of
physical robustness, using elaborate documentation, testing and other
quality control measures driven by military purchasing bureaucracies.
2. MLS systems have idiosyncratic administration tools and procedures. A
trained Unix administrator can’t just take on an MLS installation without
significant further training. A USAF survey showed that many MLS systems were installed without their features being used [1044].
3. Many applications need to be rewritten or at least greatly modified to run
under MLS operating systems [1092]. For example, compartmented mode
workstations that display information at different levels in different windows, and prevent the user from doing cut-and-paste operations from
high to low, often have problems with code which tries to manipulate the
colour map. Access to files might be quite different, as well as the format
of things like access control lists. Another source of conflict with commercial software is the licence server; if a High user invokes an application,
which goes to a licence server for permission to execute, then an MLS
operating system will promptly reclassify the server High and deny access
to Low users. So in practice, you usually end up (a) running two separate

267

268

Chapter 8

■

Multilevel Security

license servers, thus violating the license terms, or (b) you have an MLS
license server which tracks licenses at all levels (this restricts your choice
of platforms), or (c) you only access the licensed software at one of the
levels.
4. Because processes are automatically upgraded as they see new labels, the
files they use have to be too. New files default to the highest label belonging to any possible input. The result of all this is a chronic tendency for
things to be overclassified.
5. It is often inconvenient to deal with ‘blind write-up’ — when a low level
application sends data to a higher level one, BLP prevents any acknowledgment being sent. The effect is that information vanishes into a ‘black
hole’. The answer to this is varied. Some organizations accept the problem
as a fact of life; in the words of a former NSA chief scientist ‘When you
pray to God, you do not expect an individual acknowledgement of each
prayer before saying the next one’. Others use pumps rather than prayer,
and accept a residual covert bandwidth as a fact of life.
6. The classification of data can get complex:
in the run-up to a military operation, the location of ‘innocuous’ stores
such as food could reveal tactical intentions, and so may be suddenly
upgraded. It follows that the tranquility property cannot simply be
assumed;
classifications are not necessarily monotone. Equipment classified at
‘confidential’ in the peacetime inventory may easily contain components classified ‘secret’;
information may need to be downgraded. An intelligence analyst might
need to take a satellite photo classified at TS/SCI, and paste it into an
assessment for field commanders at ‘secret’. However, information
could have been covertly hidden in the image by a virus, and retrieved
later once the file is downgraded. So downgrading procedures may
involve all sorts of special filters, such as lossy compression of images
and word processors which scrub and reformat text, in the hope that
the only information remaining is that which lies in plain sight. (I will
discuss information hiding in more detail in the context of copyright
marking.)
we may need to worry about the volume of information available to
an attacker. For example, we might be happy to declassify any single
satellite photo, but declassifying the whole collection would reveal
our surveillance capability and the history of our intelligence priorities. Similarly, the government payroll may not be very sensitive per
se, but it is well known that journalists can often identify intelligence
personnel working under civilian cover from studying the evolution of

8.7 Broader Implications of MLS

departmental staff lists over a period of a few years. (I will look at this
issue — the ‘aggregation problem’ — in more detail in section 9.3.2.)
a related problem is that the output of an unclassified program acting
on unclassified data may be classified. This is also related to the aggregation problem.
7. There are always system components — such as memory management —
that must be able to read and write at all levels. This objection is dealt
with by abstracting it away, and assuming that memory management is
part of the trusted computing base which enforces our mandatory access
control policy. The practical outcome is that often a quite uncomfortably
large part of the operating system (plus utilities, plus windowing system
software, plus middleware such as database software) ends up part of the
trusted computing base. ‘TCB bloat’ constantly pushes up the cost of evaluation and reduces assurance.
8. Finally, although MLS systems can prevent undesired things (such as
information leakage) from happening, they also prevent desired things
from happening too (such as efficient ways of enabling data to be downgraded from High to Low, which are essential if many systems are to be
useful). So even in military environments, the benefits they provide can
be very questionable. The associated doctrine also sets all sorts of traps for
government systems builders. A recent example comes from the debate
over a UK law to extend wiretaps to Internet Service Providers (ISPs).
(I’ll discuss wiretapping in Part III). Opponents of the bill forced the government to declare that information on the existence of an interception
operation against an identified target would be classified ‘Secret’. This
would have made wiretaps on Internet traffic impossible without redeveloping all the systems used by Internet Service Providers to support an
MLS security policy — which would have been totally impractical. So the
UK government had to declare that it wouldn’t apply the laid down standards in this case because of cost.

8.7

Broader Implications of MLS

The reader’s reaction by this point may well be that mandatory access control
is too hard to do properly; there are just too many complications. This may be
true, and we are about to see the technology seriously tested as it’s deployed in
hundreds of millions of Vista PCs and Linux boxes. We will see to what extent
mandatory access control really helps contain the malware threat, whether
to commodity PCs or to servers in hosting centres. We’ll also see whether
variants of the problems described here cause serious or even fatal problems
for the DRM vision.

269

270

Chapter 8

■

Multilevel Security

However it’s also true that Bell-LaPadula and Biba are the simplest security
policy models we know of, and everything else is even harder. We’ll look at
other models in the next few chapters.
Anyway, although the MLS program has not delivered what was expected,
it has spun off a lot of useful ideas and know-how. Worrying about not just
the direct ways in which a secure system could be defeated but also about the
second- and third-order consequences of the protection mechanisms has been
important in developing the underlying science. Practical work on building
MLS systems also led people to work through many other aspects of computer
security, such as Trusted Path (how does a user know he’s talking to a genuine
copy of the operating system?), Trusted Distribution (how does a user know
he’s installing a genuine copy of the operating system?) and Trusted Facility
Management (how can we be sure it’s all administered correctly?). In effect,
tackling one simplified example of protection in great detail led to many things
being highlighted which previously were glossed over. The resulting lessons
can be applied to systems with quite different policies.
These lessons were set out in the ‘Rainbow Series’ of books on computer
security, produced by the NSA following the development of SCOMP and the
publication of the Orange Book which it inspired. These books are so called
because of the different coloured covers by which they’re known. The series
did a lot to raise consciousness of operational and evaluation issues that are
otherwise easy to ignore (or to dismiss as boring matters best left to the end
purchasers). In fact, the integration of technical protection mechanisms with
operational and procedural controls is one of the most critical, and neglected,
aspects of security engineering. I will have much more to say on this topic in
Part III, and in the context of a number of case studies throughout this book.
Apart from the official ‘lessons learned’ from MLS, there have been other
effects noticed over the years. In particular, the MLS program has had negative
effects on many of the government institutions that used it. There is a tactical
problem, and a strategic one.
The tactical problem is that the existence of trusted system components
plus a large set of bureaucratic guidelines has a strong tendency to displace
critical thought. Instead of working out a system’s security requirements in
a methodical way, designers just choose what they think is the appropriate
security class of component and then regurgitate the description of this class
as the security specification of the overall system [1044].
One should never lose sight of the human motivations which drive a system
design, and the costs which it imposes. Moynihan’s book [907] provides a
critical study of the real purposes and huge costs of obsessive secrecy in US
foreign and military affairs. Following a Senate enquiry, he discovered that
President Truman was never told of the Venona decrypts because the material
was considered ‘Army Property’ — despite its being the main motivation for
the prosecution of Alger Hiss. As his book puts it: ‘Departments and agencies

8.7 Broader Implications of MLS

hoard information, and the government becomes a kind of market. Secrets
become organizational assets, never to be shared save in exchange for another
organization’s assets.’ He reports, for example, that in 1996 the number of
original classification authorities decreased by 959 to 4,420 (following postCold-War budget cuts) but that the total of all classification actions reported
for fiscal year 1996 increased by 62 percent to 5,789,625.
I wrote in the first edition in 2001: ‘Yet despite the huge increase in secrecy,
the quality of intelligence made available to the political leadership appears to
have declined over time. Effectiveness is undermined by inter-agency feuding
and refusal to share information, and by the lack of effective external critique3 .
So a strong case can be made that MLS systems, by making the classification
process easier and controlled data sharing harder, actually impair operational
effectiveness’. A few months after the book was published, the attacks of 9/11
drove home the lesson that the US intelligence community, with its resources
fragmented into more than twenty agencies and over a million compartments,
was failing to join up the dots into an overall picture. Since then, massive
efforts have been made to get the agencies to share data. It’s not clear that this
is working; some barriers are torn down, others are erected, and bureaucratic
empire building games continue as always. There have, however, been leaks
of information that the old rules should have prevented. For example, a Bin
Laden video obtained prior to its official release by Al-Qaida in September
2007 spread rapidly through U.S. intelligence agencies and was leaked by
officials to TV news, compromising the source [1322].
In the UK, the system of classification is pretty much the same as the U.S.
system described in this chapter, but the system itself is secret, with the full
manual being available only to senior officials. This was a contributory factor in
a public scandal in which a junior official at the tax office wrote a file containing
the personal information of all the nation’s children and their families to two
CDs, which proceeded to get lost in the post. He simply was not aware that
data this sensitive should have been handled with more care [591]. I’ll describe
this scandal and discuss its implications in more detail in the next chapter.
So multilevel security can be a double-edged sword. It has become entrenched in government, and in the security-industrial complex generally, and
is often used in inappropriate ways. Even long-time intelligence insiders have
documented this [671]. There are many problems which we need to be a ‘fox’
rather than a ‘hedgehog’ to solve. Even where a simple, mandatory, access
control system could be appropriate, we often need to control information
flows across, rather than information flows down. Medical systems are a good
example of this, and we will look at them next.
3
Although senior people follow the official line when speaking on the record, once in private
they rail at the penalties imposed by the bureaucracy. My favorite quip is from an exasperated
British general: ‘What’s the difference between Jurassic Park and the Ministry of Defence? One’s
a theme park full of dinosaurs, and the other’s a movie!’

271

272

Chapter 8

8.8

■

Multilevel Security

Summary

Mandatory access control was developed for military applications, most
notably specialized kinds of firewalls (guards and pumps). They are being
incorporated into commodity platforms such as Vista and Linux. They have
even broader importance in that they have been the main subject of computer
security research since the mid-1970’s, and their assumptions underlie many
of the schemes used for security evaluation. It is important for the practitioner
to understand both their strengths and limitations, so that you can draw on
the considerable research literature when it’s appropriate, and avoid being
dragged into error when it’s not.

Research Problems
Multilevel confidentiality appears to have been comprehensively done to death
by generations of DARPA-funded research students. The opportunity now is
to explore what can be done with the second-generation mandatory access
control systems shipped with Vista and SELinux, and with virtualization
products such as VMware and Xen; what can be done to make it easier
to devise policies for these systems that enable them to do useful work; in
better mechanisms for controlling information flow between compartments;
the interaction which multilevel systems have with other security policies; and
in ways to make mandatory access control systems usable.
An ever broader challenge, sketched out by Earl Boebert after the NSA
launched SELinux, is to adapt mandatory access control mechanisms to safetycritical systems (see the quote at the head of this chapter, and [197]). As a tool
for building high-assurance, special-purpose devices where the consequences
of errors and failures can be limited, mechanisms such as type enforcement
and role-based access control look like they will be useful outside the world of
security. By locking down intended information flows, designers can reduce
the likelihood of unanticipated interactions.

Further Reading
The report on the Walker spy ring is essential reading for anyone interested
in the system of classifications and clearances [587]: this describes in great
detail the system’s most spectacular known failure. It brings home the sheer
complexity of running a system in which maybe three million people have a
current SECRET or TOP SECRET clearance at any one time, with a million
applications being processed each year — especially when the system was

Further Reading

designed on the basis of how people should behave, rather than on how they
actually do behave. And the classic on the abuse of the classification process
to cover up waste, fraud and mismanagement in the public sector was written
by Chapman [282].
On the technical side, one of the better introductions to MLS systems,
and especially the problems of databases, is Gollmann’s Computer Security [537]. Amoroso’s ‘Fundamentals of Computer Security Technology’ [27] is
the best introduction to the formal mathematics underlying the Bell-LaPadula,
noninterference and nondeducibility security models.
The bulk of the published papers on engineering actual MLS systems can be
found in the annual proceedings of three conferences: the IEEE Symposium on
Security & Privacy (known as ‘Oakland’ as that’s where it’s held), the National
Computer Security Conference (renamed the National Information Systems Security
Conference in 1995), whose proceedings were published by NIST until the
conference ended in 1999, and the Computer Security Applications Conference
whose proceedings are (like Oakland’s) published by the IEEE. Fred Cohen’s
experiments on breaking MLS systems using viruses are described in his book,
‘A Short Course on Computer Viruses’ [311]. Many of the classic early papers in
the field can be found at the NIST archive [934].

273

CHAPTER

9
Multilateral Security
Privacy is a transient notion. It started when people stopped
believing that God could see everything and stopped when
governments realised there was a vacancy to be filled.
— Roger Needham
You have zero privacy anyway. Get over it.
— Scott Mcnealy

9.1

Introduction

Often our goal is not to prevent information flowing ‘down’ a hierarchy but to
prevent it flowing ‘across’ between departments. Relevant applications range
from healthcare to national intelligence, and include most applications where
the privacy of individual customers’, citizens’ or patients’ data is at stake.
They account for a significant proportion of information processing systems
but their protection is often poorly designed and implemented. This has led to
a number of expensive fiascos.
The basic problem is that if you centralise systems containing sensitive
information, you risk creating a more valuable asset and simultaneously
giving more people access to it. This is now a pressing problem in the
world of ‘Web 2.0’ as online applications amass petabytes of people’s private
information. And it’s not just Google Documents; a number of organisations
plan to warehouse your medical records online. Microsoft has announced
HealthVault, which will let your doctors store your medical records online in a
data centre and give you some control over access; other IT firms have broadly
similar plans. Yet privacy activists point out that however convenient this
275

276

Chapter 9

■

Multilateral Security

may be in an emergency, it gives access to insurance companies, government
agencies and anyone else who comes along with a court order [1332]. So what
are the real issues with such systems, should they be built, if so how should
we protect them, and are there any precedents from which we can learn?
One lesson comes from banking. In the old days, a private investigator who
wanted copies of your bank statements had to subvert someone at the branch
where your account was kept. But after banks hooked all their branches up
online in the 1980s, they typically let any teller enquire about any customer’s
account. This brought the convenience of being able to cash a check when you
are out of town; but it’s also meant that private eyes buy and sell your bank
statements for a few hundred dollars. They only have to corrupt one employee
at each bank, rather than one at each branch. Another example comes from
the UK Inland Revenue, the tax collection office; staff were caught making
improper access to the records of celebrities, selling data to outsiders, and
leaking income details in alimony cases [129].
In such systems, a typical requirement will be to stop users looking at
records belonging to a different branch, or a different geographical region, or
a different partner in the firm — except under strict controls. Thus instead of
the information flow control boundaries being horizontal as we saw in the
Bell-LaPadula model as in Figure 9.1, we instead need the boundaries to be
mostly vertical, as shown in Figure 9.2.
These lateral information flow controls may be organizational, as in an
intelligence organization which wants to keep the names of agents working
in one foreign country secret from the department responsible for spying on
another. They may be privilege-based, as in a law firm where different clients’
affairs, and the clients of different partners, must be kept separate. They may
even be a mixture of the two, as in medicine where patient confidentiality
TOP SECRET
SECRET
CONFIDENTIAL
OPEN
Figure 9.1: Multilevel security

A

B

C

D

shared data
Figure 9.2: Multilateral security

E

9.2 Compartmentation, the Chinese Wall and the BMA Model

is based in law on the rights of the patient but usually enforced by limiting
medical record access to a particular hospital department.
The control of lateral information flows is a very general problem, of which
we’ll use medicine as a clear and well-studied example. The problems of
medical systems are readily understandable by the nonspecialist and have
considerable economic and social importance. Much of what we have to
say about them goes across with little or no change to the practice of other
professions, and to government applications where access to particular kinds
of classified data are restricted to particular teams or departments.
One minor problem we face is one of terminology. Information flow controls
of the type we’re interested in are known by a number of different names; in
the U.S. intelligence community, for example, they are known as compartmented
security or compartmentation. We will use the European term multilateral security
as the healthcare application is bigger than intelligence, and as the term also
covers the use of techniques such as anonymity — the classic case being deidentified research databases of medical records. This is an important part
of multilateral security. As well as preventing overt information flows, we
also have to prevent information leakage through, for example, statistical and
billing data which get released.
The use of de-identified data has wider applicability. Another example is
the processing of census data. In general, the relevant protection techniques
are known as inference control. Despite occasional differences in terminology,
the problems facing the operators of census databases and medical research
databases are very much the same.

9.2 Compartmentation, the Chinese Wall
and the BMA Model
There are (at least) three different models of how to implement access controls
and information flow controls in a multilateral security model. These are
compartmentation, used by the intelligence community; the Chinese Wall
model, which describes the mechanisms used to prevent conflicts of interest
in professional practice; and the BMA model, developed by the British Medical
Association to describe the information flows permitted by medical ethics.
Each of these has potential applications outside its initial field.

9.2.1 Compartmentation and the Lattice Model
For many years, it has been standard practice in the United States and allied
governments to restrict access to information by the use of codewords as well as
classifications. The best documented example is the codeword Ultra in World

277

278

Chapter 9

■

Multilateral Security

War 2, which referred to British and American decrypts of German messages
enciphered using the Enigma cipher machine. The fact that the Enigma had
been broken was so important that it was worth protecting at almost any cost.
So Ultra clearances were given to only a small number of people — in addition
to the cryptanalysts and their support staff, the list included the Allied leaders,
their senior generals, and hand-picked analysts. No-one who had ever held an
Ultra clearance could be placed at risk of capture; and the intelligence could
never be used in such a way as to let Hitler suspect that his principal cipher
had been broken. Thus when Ultra told of a target, such as an Italian convoy
to North Africa, the Allies would send over a plane to ‘spot’ it and report its
position by radio an hour or so before the attack. This policy was enforced
by special handling rules; for example, Churchill got his Ultra summaries in
a special dispatch box to which he had a key but his staff did not. Because
such special rules may apply, access to a codeword is sometimes referred to
as an indoctrination rather than simply a clearance. (Ultra security is described
in Kahn [677] and in Welchman [1336].)
Much the same precautions are in place today to protect information
whose compromise could expose intelligence sources or methods, such as
agent names, cryptanalytic successes, the capabilities of equipment used
for electronic eavesdropping, and the performance of surveillance satellites.
The proliferation of codewords results in a large number of compartments,
especially at classification levels above Top Secret.
One reason for this is that classifications are inherited by derived work;
so a report written using sources from ‘Secret Desert Storm’ and ‘Top Secret
Umbra’ can in theory only be read by someone with a clearance of ‘Top Secret’
and membership of the groups ‘Umbra’ and ‘Desert Storm’. Each combination
of codewords gives a compartment, and some intelligence agencies have
over a million active compartments. Managing them is a significant problem.
Other agencies let people with high level clearances have relatively wide
access. But when the control mechanisms fail, the result can be disastrous.
Aldritch Ames, a CIA officer who had accumulated access to a large number of
compartments by virtue of long service and seniority, and because he worked
in counterintelligence, was able to betray almost the entire U.S. agent network
in Russia.
Codewords are in effect a pre-computer way of expressing access control
groups, and can be dealt with using a variant of Bell-LaPadula, called the lattice
model. Classifications together with codewords form a lattice — a mathematical
structure in which any two objects A and B can be in a dominance relation
A > B or B > A. They don’t have to be: A and B could simply be incomparable
(but in this case, for the structure to be a lattice, they will have a least upper
bound and a greatest lower bound). As an illustration, suppose we have
a codeword, say ‘Crypto’. Then someone cleared to ‘Top Secret’ would be
entitled to read files classified ‘Top Secret’ and ‘Secret’, but would have no

9.2 Compartmentation, the Chinese Wall and the BMA Model

access to files classified ‘Secret Crypto’ unless he also had a crypto clearance.
This can be expressed as shown in Figure 9.3.
In order for information systems to support this, we need to distill the
essence of classifications, clearances and labels into a security policy that we
can then use to drive security targets, implementation, and evaluation. As it
happens, the Bell-LaPadula model goes across more or less unchanged. We
still have information flows between High and Low as before, where High is
a compartment that dominates Low. If two nodes in a lattice are incompatible — as with ‘Top Secret’ and ‘Secret Crypto’ in the above diagram — then
there should be no information flow between them at all.
In fact, the lattice and Bell-LaPadula models are essentially equivalent, and
were developed at the same time.
Roger Schell, Peter Downey, and Gerald Popek of the U.S. Air Force produced an early lattice model in 1972 [1119].
A Cambridge PhD thesis by Jeffrey Fenton included a representation in
which labels were managed using a matrix [464].
About this time, the Pentagon’s World Wide Military Command and
Control System (WWMCCS) used a primitive lattice model, but without
the *-property. The demonstration that a fielded, critical, system handling Top Secret data was vulnerable to attack by Trojans caused some
consternation [1118]. It meant that all users had to be cleared to the highest level of data in the machine.
(TOP SECRET, {CRYPTO, FOREIGN})
(TOP SECRET, {CRYPTO})

(TOP SECRET, {})
(SECRET, {CRYPTO, FOREIGN})

(SECRET, {CRYPTO})
(SECRET, {})

(UNCLASSIFIED, {})

Figure 9.3: A lattice of security labels

279

280

Chapter 9

■

Multilateral Security

Kenneth Walter, Walter Ogden, William Rounds, Frank Bradshaw, Stan
Ames, and David Shumway of Case Western University produced a
more advanced lattice model as well as working out a lot of the problems
with file and directory attributes, which they fed to Bell and LaPadula
[1312, 1313]1 .
Finally, the lattice model was systematized and popularized by
Denning [368].
Most products built for the multilevel secure market can be reused in
compartmented mode. But, in practice, these products are not as effective as
one might like. It is easy to use a multilevel operating system to keep data in
different compartments separate — just give them incompatible labels (‘Secret
Tulip’, ‘Secret Daffodil’, ‘Secret Crocus’, . . .). But the operating system has
now become an isolation mechanism, rather than a sharing mechanism; the
real problem is how to control information sharing.
One solution is to impose least upper bounds in the lattice using some
algorithm. An example comes from the system used by the government of
Saudi Arabia to manage the Haj, the annual pilgrimage to Mecca [606]. While
most compartments are by default Confidential, the combination of data
from different compartments is Secret. Thus ‘Haj-visas’ and ‘Gov-guest’ are
confidential, but their combination is Secret.
In many intelligence systems, where the users are already operating at the
highest level of clearance, data owners don’t want a further classification level
at which everything is visible. So data derived from two compartments effectively creates a third compartment using the lattice model. The proliferation
of millions of compartments is complex to manage and can be intertwined
with applications. So a more common solution is to use a standard multilevel
product, such as a mail guard, to ensure that ‘untrustworthy’ email goes to
filters. But now the core of the trusted computing base consists of the filters
rather than the guard.
Worse, the guard may lose some of the more important functionality of the
underlying operating system. For example, the Standard Mail Guard [1193]
was built on top of an operating system called LOCK whose basic mechanism is type enforcement, as described in the previous chapter. Later
versions of LOCK support role-based access control, which would be a more
appropriate mechanism to manage the relationships between compartments
directly [612]. Using it merely as a platform to support BLP may have been
wasteful.
In general, the real problems facing users of intelligence systems have to
do with combining data in different compartments, and downgrading it after
1
Walter and his colleagues deserve more credit than history has given them. They had the
main results first [1312] but Bell and LaPadula had their work heavily promoted by the U.S.
Air Force. Fenton has also been largely ignored, not being an American.

9.2 Compartmentation, the Chinese Wall and the BMA Model

sanitization. Multilevel and lattice security models offer little help here. Indeed
one of the biggest problems facing the U.S. intelligence community since 9/11
is how to handle search over systems with many compartments. A search done
over many agencies’ databases can throw up results with many codewords
attached; if this were to be aggregated in one place, then that place would in
effect possess all clearances. What new systems do is to send out search queries
bound with the clearance of the user: ‘Show me everything that matches Uzbek
and Peshawar and weapons and motorcycle, and can be seen by someone with
a clearance of Top Secret Umbra’. Here, local labels just get in the way; but
without them, how do you forestall a future Aldritch Ames?
There’s a also sobering precedent in the Walker spy case. There, an attempt
to keep naval vessels in compartments just didn’t work, as a ship could be sent
anywhere on no notice, and for a ship to be isolated with no local key material
was operationally unacceptable. So the U.S. Navy’s 800 ships all ended up
with the same set of cipher keys, which got sold to the Russians [587].

9.2.2 The Chinese Wall
The second model of multilateral security is the Chinese Wall model, developed
by Brewer and Nash [224]. Its name comes from the fact that financial services
firms from investment banks to accountants have internal rules designed to
prevent conflicts of interest, which they call Chinese Walls.
The model’s scope is wider than just finance. There are many professional
and services firms whose clients may be in competition with each other:
software vendors and advertising agencies are other examples. A typical rule
is that ‘a partner who has worked recently for one company in a business
sector may not see the papers of any other company in that sector’. So once an
advertising copywriter has worked on (say) the Shell account, he will not be
allowed to work on any other oil company’s account for some fixed period of
time.
The Chinese Wall model thus features a mix of free choice and mandatory
access control: a partner can choose which oil company to work for, but once
that decision is taken his actions in that sector are completely constrained. It
also introduces the concept of separation of duty into access control; a given
user may perform transaction A or transaction B, but not both.
Part of the attraction of the Chinese Wall model to the security research
community comes from the fact that it can be expressed in a way that is fairly
similar to Bell-LaPadula. If we write, for each object c, y(c) for c’s company and
x(c) for c’s conflict-of-interest class, then like BLP it can be expressed in two
properties:
The simple security property: a subject s has access to c if and only if, for all
/ x(c ) or y(c) = y(c )
c which s can read, either y(c) ∈

281

282

Chapter 9

■

Multilateral Security

The *-property: a subject s can write to c only if s cannot read any c with
x(c ) =  and y(c) = y(c ).
The Chinese Wall model made a seminal contribution to the theory of access
control. It also sparked a debate about the extent to which it is consistent
with the BLP tranquility properties, and some work on the formal semantics
of such systems (see, for example, Foley [480] on the relationship with noninterference). There are also some interesting new questions about covert
channels. For example, could an oil company find out whether a competitor
which used the same investment bank was planning a bid for a third oil
company, by asking which specialists were available for consultation and
noticing that their number had dropped suddenly?
In practice, however, Chinese Walls still get implemented using manual
methods. One large software consultancy has each of its staff maintain an
‘unclassified’ curriculum vitae containing entries that have been sanitized and
agreed with the customer. A typical entry might be:
Sep 97 — Apr 98: consulted on security requirements for a new branch
accounting system for a major U.S. retail bank
This is not the only control. A consultant’s manager should be aware of
possible conflicts and not forward the CV to the client if in doubt; if this fails
the client can spot potential conflicts himself from the CV; and if this also fails
then the consultant is duty bound to report any potential conflicts as soon as
they appear.

9.2.3

The BMA Model

Perhaps the most important, interesting and instructive example of multilateral security is found in medical information systems. The healthcare sector
spends a much larger share of national income than the military in developed
countries, and although hospitals are still less automated, they are catching
up fast. A 2006 study for the U.S. Department of Health and Human Services
(DHHS) showed that investments in health IT were recouped in from three to
thirteen years, and could make health care safer as well as more efficient [1160].
Healthcare safety and (especially) privacy have become hot-button issues
in many countries. In the USA, the Health Insurance Portability and Accountability Act (HIPAA) was passed by Congress in 1996 following a number of
privacy failures. In one notorious case, Mark Farley, a convicted child rapist
working as an orthopedic technician at Newton-Wellesley Hospital in Newton,
Massachusetts, was caught using a former employee’s password to go through
the records of 954 patients (mostly young females) to get the phone numbers of
girls to whom he then made obscene phone calls [223]. He ended up doing jail
time, and the Massachusetts senator Edward Kennedy was one of HIPAA’s

9.2 Compartmentation, the Chinese Wall and the BMA Model

sponsors. There are many more incidents of a less dramatic nature. Also
in 1995–96, the UK government attempted to centralise all medical records,
which led to a confrontation with the British Medical Association (BMA). The
BMA hired me to devise a policy for safety and privacy of clinical information,
which I’ll discuss below.
The controversy continued. In the late 1990s, a project in Iceland to build
a national medical database incorporating not just medical records but also
genetic and genealogical data, so that inherited diseases can be tracked across
generations, caused an uproar. Eleven percent of the population opted out;
eventually the Icelandic Supreme Court decided that the database had to be
opt-in rather than opt-out, and now about half the population participate.
In 2002, President Bush rewrote and relaxed the HIPAA regulations, known
as the ‘Privacy Rule’; this was followed by further ‘administrative simplification’ in 2006. The U.S. situation is now that, although medical data must still be
protected in hospitals, clinics and insurers, its use outside the immediate care
setting (for example, by researchers, employers and welfare agencies) is outside the regulations and so much less controlled. No-one’s completely happy:
health privacy advocates consider the regime to be quite inadequate; hospitals
complain that it adds unnecessarily to their costs; and patient advocates note
that HIPAA is often used by hospital staff as an excuse to be unhelpful [560].
At the time of writing (2007), Atlanta’s Piedmont Hospital has just become
the first institution in the USA to be audited for compliance with the security
and privacy regulations, which came into force in 2005. This audit covered
topics from physical and logical access to systems and data through Internet
usage to violations of security rules by employees, and helped many other
healthcare providers decide to invest in encryption and other protection technologies [1295]. In addition, the Government Accountability Office (GAO) has
just reported that the DHHS needs to do a lot more to ensure patient privacy,
particularly by defining an overall strategy for privacy and by adopting milestones for dealing with nationwide health data exchange (which is not just a
matter of inadequate technical protection but also of varying state laws) [735].
In various European countries, there have been debates about the safety
and privacy tradeoffs involved with emergency medical information. The
Germans put data such as current prescriptions and allergies on the medical
insurance card that residents carry; other countries have held back from this,
reasoning that if data currently held on a human-readable MedAlert bracelet,
such as allergies, are moved to a machine-readable device such as a smartcard,
then there is a risk to patients who fall ill in locations where there is no
reader available, such as on an airplane or a foreign holiday. In the UK, the
government is creating a ‘summary care record’ of prescriptions and allergies
that will be kept on a central database and will be available to many health-care
workers, from emergency room clinicians to paramedics and the operators of
out-of-hours medical helpline services. One problem is that a patient’s current

283

284

Chapter 9

■

Multilateral Security

medications often reveal highly sensitive information — such as treatment for
HIV, depression or alcoholism — and making such information available to
hundreds of thousands of people carries substantial risks of abuse. Patients
have been offered the right to opt out of this system.
There have also been debates about privacy and ethical issues relating to
secondary uses of medical information, such as in research. First, there are
worries about privacy failures, for example, when a research professor loses a
laptop containing the records of millions of patients. Although records used in
research often have names and addresses removed, it is a seriously hard job to
de-identify records properly; I’ll discuss this in detail below. Second, there are
ethics issues related to consent. For example, a devout Catholic woman might
object to her gynaecological data being used to develop a better morning-after
pill. Third, there are economic issues; if my data get used to develop a drug
from which a company makes billions of dollars, shouldn’t I get a share?
The protection of medical information is thus an interesting case history for
the security engineer. It has a lot of rich and complex tradeoffs; it’s important
to all of us; and it’s frequently in the news.
Medical privacy is also a model for protecting personal information of other
kinds, such as the information held on individual customers by companies
and government agencies. In all European countries (and in many others,
such as Canada and Australia) there are data protection laws that restrict
the dissemination of such data. I’ll discuss data protection law in Part III; for
present purposes, it’s enough to note that some classes of data (affecting health,
sexual behavior, political activity and religious belief) the data subject must
either consent to information sharing, or have a right of veto, or there must be
a specific law that permits sharing for the public interest in circumstances that
are well enough defined for the data subject to predict them. This raises the
issue of how one can construct a security policy in which the access control
decisions are taken not by a central authority (as in Bell-LaPadula) or by the
system’s users (as in discretionary access control) but by the data subjects.
Let’s look first at the access control aspects.

9.2.3.1

The Threat Model

The main threat to medical privacy is abuse of authorised access by insiders,
and the most common threat vector is social engineering. The typical attack
comes from a private detective who phones a doctor’s office or health insurer
with a plausible tale:
Hello, this is Dr Burnett of the cardiology department at the Conquest
Hospital in Hastings. Your patient Sam Simmonds has just been admitted here in a coma, and he has a funny looking ventricular arrhythmia. Can you tell me if there’s anything relevant in his record?

9.2 Compartmentation, the Chinese Wall and the BMA Model

This kind of attack is usually so successful that in both the USA and the
UK there are people who earn their living doing it [411]. (It’s not restricted to
health records — in June 2000, Tony Blair’s fundraiser Lord Levy was acutely
embarrassed after someone called the tax office pretending to be him and
found out that he’d only paid £5000 in tax the previous year [1064]. But the
medical context is a good one in which to discuss it.)
As I mentioned briefly in Chapter 2, an experiment was done in the UK
in 1996 whereby the staff at a health authority (a government-owned insurer
that purchases health care for a district of several hundred thousand people)
were trained to screen out false-pretext telephone calls. The advice they were
given is described in [36] but the most important element of it was that they
were to always call back — and not to a number given by the caller, but to the
number in the phone book for the hospital or other institution where the caller
claimed to work. It turned out that some thirty telephone enquiries a week
were bogus.
Such operational security measures are much more important than most
technical protection measures, but they are difficult. If everyone was as
unhelpful as intelligence-agency staff are trained to be, the world would grind
to a halt. And the best staff training in the world won’t protect a system where
too many people see too much data. There will always be staff who are careless
or even crooked; and the more records they can get, the more harm they can
do. Also, organisations have established cultures; we have been simply unable
to embed even lightweight operational-security measures on any scale in
healthcare, simply because that’s not how people work. Staff are focussed on
delivering care rather than questioning each other. The few real operational
improvements in the last few years have all followed scares; for example,
maternity units in Britain now have reasonable entry controls, following
incidents in which babies were stolen from nurseries. Also, geriatric wards are
often locked to stop demented patients from wandering off. However, most
hospital wards are completely open; anyone can wander in off the street to
visit their relatives, and the clinical benefits of frequent visits outweigh the
occasional violent incidents. PCs are left unattended and logged on to the
hospital network. Recently, a health IT investment programme in the UK has
tried to standardise access control and issued clinical staff with smartcards to
log on to hospital systems; but since logging off as Nurse Jones and on again
as Nurse Smith takes several seconds, staff don’t bother.
A more general problem is that even where staff behave ethically, a lack
of technical understanding — or, as we might more properly describe it, poor
security usability — causes leaks of personal information. Old PCs sold on the
second hand market or given to schools often have recoverable data on the
hard disk; most people are unaware that the usual ‘delete’ command does
not remove the file, but merely marks the space it occupies as re-usable. A
PC sold on the second hand market by investment bank Morgan Grenfell

285

286

Chapter 9

■

Multilateral Security

Asset Management had recoverable files containing the financial dealings of
ex-Beatle Paul McCartney [254]: there have been similar problems with old
health records. Equipment also gets stolen: some 11% of UK family doctors
have experienced the theft of a practice PC, and in one case two prominent
society ladies were blackmailed over terminations of pregnancy following such
a theft [37]. The UK government response to this threat is to try to persuade
family doctors to move to ‘hosted’ systems, where the practice data are kept
on regional server farms; but it’s quite unclear that there’s a net privacy gain.
Data theft may be harder, but once data are centralised you can expect access
creep; more and more public agencies will come up with arguments why they
need access to the data. Even if all the access cases are individually sound, the
net effect over time can be quite destructive of privacy.
The fundamental problem is this. The likelihood that a resource will be
abused depends on its value and on the number of people who have access to
it. Aggregating personal information into large databases increases both these
risk factors at the same time. Put simply, we can live with a situation in which
a doctor’s receptionist has access to 2,000 patients’ records: there will be abuse
from time to time, but at a tolerably low level. However, if the receptionists
of the 5,000 family doctors who might work with a large American HMO,
or in one of the five regions of England’s National Health Service, all have
access to the records of maybe ten million patients, then abuse becomes likely.
It only takes one insider who learns to walk up to a PC that’s logged on
using someone else’s smartcard, read a file, and pass the information on to a
private eye in exchange for cash. It’s not just doctors; in England, each region
has tens of thousands of people with access, from nurses and programmers
and receptionists to drivers and caterers and cleaners. Many of the staff are
temporary, many are foreign, and many are earning close to the minimum
wage. And privacy issues aren’t limited to organizations that treat patients
directly: some of the largest collections of personal health information are
in the hands of health insurers and research organizations. I’ll discuss their
special problems below in section 9.3.
In such an environment, lateral information flow controls are required. A
good example of what can go wrong without them comes from an early UK
hospital system whose designers believed that for reasons of safety, all staff
should have access to all records. This decision was influenced by lobbying
from geriatricians and pediatricians, whose patients are often treated by a
number of specialist departments in the hospital. They were frustrated by
the incompatibilities between different departmental systems. The system was
fielded in 1995 in Hampshire, where the then health minister Gerry Malone
had his parliamentary seat. The system made all lab tests performed for local
doctors at the hospital’s pathology lab visible to most of the hospital’s staff.
A nurse who had had a test done by her family doctor complained to him
after she found the result on the hospital system at Basingstoke where she

9.2 Compartmentation, the Chinese Wall and the BMA Model

worked; this caused outrage among local medics, and Malone lost his seat in
Parliament at the 1997 election (by two votes) [46].
So how can we avoid letting everyone see every record? There are many
ad-hoc things you can do: one fairly effective measure is to keep the records
of former patients in a separate archive, and give only a small number of
admissions staff the power to move records from there to the main system.
Another is to introduce a honey trap: one Boston hospital has on its system
some bogus ‘medical records’ with the names of Kennedy family members, so
it can identify and discipline staff who browse them. A particularly ingenious
proposal, due to Gus Simmons, is to investigate all staff who consult a patient
record but do not submit a payment claim to the insurer within thirty days;
this aligns the patient’s interest in privacy with the hospital’s interest in
maximizing its income.
However, a patchwork of ad-hoc measures isn’t a good way to secure a
system. We need a proper access control policy, thought through from first
principles and driven by a realistic model of the threats. What policy is
appropriate for healthcare?

9.2.3.2

The Security Policy

This question faced the BMA in 1995. The UK government had introduced
an IT strategy for the National Health Service which involved centralizing a
lot of data on central servers and whose security policy was multilevel: the
idea was that AIDS databases would be at a level corresponding to Secret,
normal patient records at Confidential and administrative data such as drug
prescriptions and bills for treatment at Restricted. It was soon realised that
this wasn’t going to work. For example, how should a prescription for AZT
be classified? As it’s a drug prescription, it should be Restricted; but as it
identifies a person as HIV positive, it must be Secret. So all the ‘Secret’ AZT
prescriptions must be removed from the ‘Restricted’ file of drug prescriptions.
But then so must almost all the other prescriptions as they identify treatments
for named individuals and so should be ‘Confidential’. But then what use will
the file of prescriptions be to anybody?
A second problem is that the strategy was based on the idea of a single
electronic patient record (EPR) that would follow the patient around from
conception to autopsy, rather than the traditional system of having different
records on the same patient at different hospitals and doctors’ offices, with
information flowing between them in the form of referral and discharge
letters. An attempt to devise a security policy for the EPR, which would
observe existing ethical norms, quickly became unmanageably complex [558].
In a project for which I was responsible, the BMA developed a security
policy to fill the gap. The critical innovation was to define the medical record
not as the total of all clinical facts relating to a patient, but as the maximum

287

288

Chapter 9

■

Multilateral Security

set of facts relating to a patient and to which the same staff had access. So
an individual patient will have more than one record, and this offended the
‘purist’ advocates of the EPR. But multiple records are dictated anyway by law
and practice. Depending on the country (and even the state) that you’re in, you
may have to keep separate medical records for human fertilization, sexually
transmitted diseases, prison medical services, and even birth records (as they
pertain to the health of the mother as well as the child, and can’t simply be
released to the child later without violating the mother’s confidentiality). This
situation is likely to get more complex still as genetic data start being used
more widely.
In many countries, including all signatories to the European Convention on
Human Rights, a special status is given to patient consent in law as well as
in medical ethics. Records can only be shared with third parties if the patient
approves, or in a limited range of statutory exceptions, such as tracing contacts
of people with infectious diseases like TB. Definitions are slightly fluid; in
some countries, HIV infection is notifiable, in others it isn’t, and in others the
data are collected stealthily.
The goals of the BMA security policy were therefore to enforce the principle
of patient consent, and to prevent too many people getting access to too
many identifiable records. It did not try to do anything new, but merely to
codify existing best practice. It also sought to express other security features of
medical record management such as safety and accountability. For example,
it must be possible to reconstruct the contents of the record at any time in the
past, so that for example if a malpractice suit is brought the court can determine
what information was available to the doctor at the time. The details of the
requirements analysis are in [37].
The policy consists of nine principles.
1. Access control: each identifiable clinical record shall be marked with an
access control list naming the people or groups of people who may read
it and append data to it. The system shall prevent anyone not on the access
control list from accessing the record in any way.
2. Record opening: a clinician may open a record with herself and the patient
on the access control list. Where a patient has been referred, she may
open a record with herself, the patient and the referring clinician(s) on the
access control list.
3. Control: One of the clinicians on the access control list must be marked as
being responsible. Only she may alter the access control list, and she may
only add other health care professionals to it.
4. Consent and notification: the responsible clinician must notify the patient
of the names on his record’s access control list when it is opened, of all
subsequent additions, and whenever responsibility is transferred. His

9.2 Compartmentation, the Chinese Wall and the BMA Model

consent must also be obtained, except in emergency or in the case of statutory exemptions.
5. Persistence: no-one shall have the ability to delete clinical information
until the appropriate time period has expired.
6. Attribution: all accesses to clinical records shall be marked on the record
with the subject’s name, as well as the date and time. An audit trail must
also be kept of all deletions.
7. Information flow: Information derived from record A may be appended to
record B if and only if B’s access control list is contained in A’s.
8. Aggregation control: there shall be effective measures to prevent the
aggregation of personal health information. In particular, patients must
receive special notification if any person whom it is proposed to add to
their access control list already has access to personal health information
on a large number of people.
9. Trusted computing base: computer systems that handle personal health
information shall have a subsystem that enforces the above principles in
an effective way. Its effectiveness shall be subject to evaluation by independent experts.
This policy may seem to be just common sense, but is surprisingly comprehensive and radical in technical terms. For example, it is strictly more
expressive than the Bell-LaPadula model of the last chapter; it contains a
BLP-type information flow control mechanism in principle 7, but also contains
state. (A fuller discussion from the point of view of access control, and for a
technical audience, can be found at [38].)
Similar policies were developed by other medical bodies including the
Swedish and German medical associations; the Health Informatics Association
of Canada, and an EU project (these are surveyed in [732]). However the BMA
model is the most detailed and has been subjected to the most rigorous
review; it was adopted by the Union of European Medical Organisations
(UEMO) in 1996. Feedback from public consultation on the policy can be
found in [39].

9.2.3.3

Pilot Implementations

In a top-down approach to security engineering, one should first determine
the threat model, then write the policy, and then finally test the policy by
observing whether it works in real life.
BMA-compliant systems have now been implemented both in general
practice [585], and in a hospital system developed in Hastings, England, that
enforces similar access rules using a mixture of roles and capabilities. It has
rules such as ‘a ward nurse can see the records of all patients who have within

289

290

Chapter 9

■

Multilateral Security

the previous 90 days been on her ward’, ‘a doctor can see the records of all
patients who have been treated in her department’, and ‘a senior doctor can
see the records of all patients, but if she accesses the record of a patient who has
never been treated in her department, then the senior doctor responsible for
that patient’s care will be notified’. (The hospital system was initially designed
independently of the BMA project. When we learned of each other we were
surprised at how much our approaches coincided, and reassured that we had
captured the profession’s expectations in a reasonably accurate way.)
The lessons learned are discussed in [366, 367, 585]. One was the difficulty
of constructing a small trusted computing base. The hospital records system
has to rely on the patient administrative system to tell it which patients, and
which nurses, are on which ward. A different prototype system at a hospital
in Cambridge, England, furnished staff with certificates in smartcards which
they used to log on.

9.2.4

Current Privacy Issues

In 2002, Prime Minister Tony Blair was persuaded to allocate £6bn to modernise
health service computing in England. This led to a scramble for contracts with
security being something of an afterthought. The original vision was for
much improved communications in each local health community; so that if a
diabetic patient was being seen by a family doctor, a hospital diabetologist,
a community nurse and an optician, they would all be able to see each others’
notes and test results. The patient herself would also be able to upload data
such as blood glucose levels, see her medical notes, and participate in her care.
This vision had been pioneered in the Wirral near Liverpool.
When the dust of the contracting process had settled, the local empowerment
vision had been replaced with a much more central approach. Contracts were
let for five regions, each with about 10 million people, calling for all hospital
systems to be replaced during 2004–2010 with standard ones. The number of
system suppliers has been whittled down to two — Cerner and iSoft — and
the security policy has been the subject of much debate. The current policy is
for three main mechanisms.
1. The workhorse of access control will be role-based access controls, similar to those pioneered at Hastings, but much more complex; rather than a
dozen or so roles the plan is now for there to be over three hundred.
2. In order to access patient data, a staff member will also need a legitimate
relationship. This is an abstraction of the Hastings idea of ‘her department’.
3. By default each patient has a single electronic patient record. However,
patients will also be able to declare that certain parts of their records are
either ‘sealed’ or ‘sealed and locked’. In the latter case, the records will
only be visible to a particular care team. In the former, their existence will

9.2 Compartmentation, the Chinese Wall and the BMA Model

be visible to other staff who look at the patient record, and who will be
able to break the seal in an emergency.
Initial implementations have thrown up a whole host of detailed problems.
For example, patients receiving outpatient psychiatric care at a hospital used
to have their notes kept in paper in the psychiatrist’s filing cabinet; all the
receptionist got to know was that Mrs Smith was seen once a month by
Dr Jones. Now, however, the receptionist can see the notes too. Her role
had to be given access to patient records so that she could see and amend
administrative data such as appointment times; and if she’s working reception
in the hospital wing where Dr Jones has his office, then she has a legitimate
relationship. Record sealing and locking aren’t implemented yet. Thus she
gets access to everything. This is a good example of why the ‘EPR’ doctrine of
one record per patient was a bad idea, and the BMA vision of multiple linked
records was better; it now looks like all records in psychiatry, sexual health etc
may have to be sealed (or even sealed-and-locked) by default. Then the care of
such patients across different departments will start to cause problems. As with
multilevel secure systems, the hard thing isn’t so much separating systems,
but managing information flows across levels, or across compartments.
Perhaps the toughest problems with the new English systems, however,
concern patient consent. The health service is allowing people to opt out of the
summary care record — the central database of emergency medical information, containing things like medications, allergies and major medical history.
This is not such a big deal; most people have nothing stigmatising in there.
(Indeed, most people under the retirement age have no significant chronic
conditions and could do perfectly well without a summary record.) The bigger
deal is that the new hospital systems will make detailed records available to
third parties as never before, for research, health service management and
even law enforcement.
Previously, your medical privacy was protected by the fact that a hospital
might have had over seventy different departmental record systems, while
your records at your family doctor were protected by being partly on paper
and partly on a PC that was switched off at six every evening and to which
outsiders had no access. Once everything sits in standard systems on a regional
health server farm, the game changes. Previously, a policeman who wanted to
see your medical records needed to persuade a judge that he had reasonable
grounds to believe he would find actual evidence of a crime; he then had
to take the warrant along to your family doctor, or your hospital’s medical
director. The costs of this procedure ensured that it was invoked only rarely,
and in cases like terrorism, murder or rape. A server farm, though, is a much
easier target — and if it contains data of everyone who’s confessed illegal drug
use to their doctor, it’s a tempting target. Indeed, from June 2007 all UK doctors
are supposed to complete a ‘treatment outcomes profile’ for drug users, asking

291

292

Chapter 9

■

Multilateral Security

them whether they’ve committed any crimes in the past four weeks, including
theft, assault and selling drugs. It’s hard to believe that this information won’t
eventually find its way into police hands. But what are the consequences for
public health when people can no longer trust their doctors — especially the
most vulnerable and marginalised members of society? We already have cases
of immigrants with TB absconding, since health service demographic data
started being used to find illegal immigrants.
Thus even if the security policy in centralised systems amounts to a faithful
implementation of the BMA policy — with the exception of the eighth principle
of non-aggregation — we may expect problems. There are some aspects of
security policy that just don’t scale. Creating large databases of sensitive
personal information is intrinsically hazardous. It increases the motive for
abuse, and the opportunity for abuse, at the same time. And even if the
controls work perfectly to prevent unlawful abuse (whether by outsiders or
insiders) the existence of such databases can lead to lawful abuse — powerful
interests in society lobby for, and achieve, access to data on a scale and of a
kind that sensible people would not permit.
There are some advantages to standard central systems. In the USA, the
Veterans’ Administration runs such systems for its hospital network; after
Hurricane Katrina, veterans from Louisiana who’d ended up as refugees in
Texas or Florida, or even Minnesota, could go straight to local VA hospitals and
find their notes there at the doctor’s fingertips. Patients of many other hospitals
and clinics in New Orleans lost their notes altogether. But centralization can
definitely harm privacy. In May 2006, the personal information on all 26.5
million U.S. veterans — including names, social security numbers and in some
cases disabilities — was stolen from the residence of a Department of Veterans
Affairs employee who had taken the data home without authorization. And
it’s not enough just to compartmentalise the medical records themselves: in
the Netherlands, which has carefully avoided record centralization, there is
still a ‘Vecozo’ database that contains medical insurance details on citizens,
and almost 80,000 people had access to it, from doctors and pharmacists to
alternative healers and even taxi firms. There was a scandal when journalists
found it was easy to get the private addresses and ex-directory phone numbers
of a number of famous politicians, criminals and personalities [126]. (After the
scandal broke, the insurers and their database operator each tried to blame
the other — neither would accept responsibility for the fact that it made too
much information available to too many people.)
So if a political decision is taken to have a large centralised database, the
aggregation issue will haunt the detailed design and continued operation:
even if some people (or applications) are allowed to look at everything, it’s
an extremely bad idea not to control the principals that actually do so. If you
find that most physicians at your hospital look at a few thousand out of the
several million records in the database, and one looks at all of them, what does

9.3 Inference Control

that tell you? You’d better find out2 . But many fielded systems don’t have rate
controls, or effective alarms, and even where alarms exist they are often not
acted on. Again in the UK, over 50 hospital staff looked at the records of a
footballing personality in hospital, despite not being involved in his care, and
none of them was disciplined.
And even apart from controversial uses of medical records, such as police
access, there are serious problems in protecting relatively uncontroversial uses,
such as research. I’ll turn to that next.

9.3

Inference Control

Access control in medical record systems is hard enough in hospitals and
clinics that care for patients directly. It is much harder to assure patient
privacy in secondary applications such as databases for research, cost control
and clinical audit. This is one respect in which doctors have a harder time
protecting their data than lawyers; lawyers can lock up their confidential client
files and never let any outsider see them at all, while doctors are under all
sorts of pressures to share data with third parties.

9.3.1 Basic Problems of Inference Control in Medicine
The standard way of protecting medical records used in research is to remove
patients’ names and addresses and thus make them anonymous. Indeed,
privacy advocates often talk about ‘Privacy Enhancing Technologies’ (PETs)
and de-identification is a frequently cited example. But this is rarely bulletproof. If a database allows detailed queries, then individuals can still usually
be identified, and this is especially so if information about different clinical
episodes can be linked. For example, if I am trying to find out whether a
politician born on the 2nd June 1946 and treated for a broken collar bone after
a college football game on the 8th May 1967, had since been treated for drug
or alcohol problems, and I could make an enquiry on those two dates, then
I could very probably pull out his record from a national database. Even if
the date of birth is replaced by a year of birth, I am still likely to be able to
compromise patient privacy if the records are detailed or if records of different
individuals can be linked. For example, a query such as ‘show me the records
of all women aged 36 with daughters aged 14 and 16 such that the mother and
exactly one daughter have psoriasis’ is also likely to find one individual out of
2
In November 2007, a former DuPont scientist was sentenced for theft of trade secrets after they
noticed he was downloading more internal documents than almost anyone else in the firm, and
investigated [294]. It’s not just hospitals and spooks that need to keep an eye on data aggregation!

293

294

Chapter 9

■

Multilateral Security

millions. And complex queries with lots of conditions are precisely the kind
that researchers want to make.
For this reason, the U.S. Healthcare Finance Administration (HCFA), which
pays doctors and hospitals for treatments provided under the Medicare
program, maintains three sets of records. There are complete records, used
for billing. There are beneficiary-encrypted records, with only patients’ names
and social security numbers obscured. These are still considered personal data
(as they still have dates of birth, postal codes and so on) and so are only
usable by trusted researchers. Finally there are public-access records which
have been stripped of identifiers down to the level where patients are only
identified is general terms such as ‘a white female aged 70–74 living in
Vermont’. Nonetheless, researchers have found that many patients can still
be identified by cross-correlating the public access records with commercial
databases, and following complaints by privacy advocates, a report from the
General Accounting Office criticised HCFA for lax security [520].
U.S. law, which comes under the HIPAA privacy rule, now recognizes
de-identified information as medical data that has been ‘properly’ de-identified.
This means either that 18 specific identifiers have been removed and the
database operator has no actual knowledge that the remaining information can
be used alone or in combination with other data to identify the subject; or that a
qualified statistician concludes that the risk is substantially limited. Where such
data are inadequate for research, it also recognises limited data sets that contain
more information, but where the users are bound by contractual and technical
measures to protect the information and not to try to re-identify subjects.
Many other countries have healthcare monitoring systems that use similar
approaches. Germany has very strict privacy laws and takes the ‘de-identified
information’ route; the fall of the Berlin Wall forced the former East German
cancer registries to install protection mechanisms rapidly [192]. New Zealand
takes the ‘limited data sets’ approach with a national database of encryptedbeneficiary medical records; access is restricted to a small number of specially
cleared medical statisticians, and no query is answered with respect to less
than six records [955]. In Switzerland, some research systems were replaced at
the insistence of privacy regulators [1137].
In other countries, protection has been less adequate. Britain’s National
Health Service built a number of centralized databases in the 1990s that
make personal health information widely available within government and
that led to confrontation with doctors. The government set up a committee
to investigate under Dame Fiona Caldicott; her report identified over sixty
illegal information flows within the health service [46, 252]. Some research
datasets were de-identified; others (including data on people with HIV/AIDS)
were re-identified afterwards, so that people and HIV charities whose data
had been collected under a promise of anonymity were deceived. Parliament
then passed a law giving ministers the power to regulate secondary uses of

9.3 Inference Control

medical data. Data kept for secondary uses are kept with postcode plus date
of birth, and as UK postcodes are shared by at most a few dozen houses,
this means that most records are easily identifiable. This remains a cause
of controversy. In 2007, Parliament’s Health Select Committee conducted an
inquiry into the Electronic Patient Record, and heard evidence from a wide
range of viewpoints — from researchers who believed that the law should
compel information sharing for research, through to physicians, humanrights lawyers and privacy advocates who argued that there should only be
the narrowest exceptions to medical privacy3 . The Committee made many
recommendations, including that patients should be permitted to prevent the
use of their data in research [624]. The Government rejected this.
The most controversial of all was a genetic database in Iceland, which I’ll
discuss in more detail below.
Stripping personal information is important in many other fields. Under the
rubric of Privacy Enhancing Technology (PET) it has been promoted recently
by regulators in Europe and Canada as a general privacy mechanism [447].
But, as the medical examples show, there can be serious tension between the
desire of researchers for detailed data, and the right of patients (or other data
subjects) to privacy. Anonymisation is much more fragile than it seems; and
when it fails, companies and individuals that relied on it can suffer serious
consequences.
AOL faced a storm of protest in 2006 when it released the supposedly
anonymous records of 20 million search queries made over three months
by 657,000 people. Searchers’ names and IP addresses were replaced with
numbers, but that didn’t help. Investigative journalists looked through the
searches and rapidly identifid some of the searchers, who were shocked at the
privacy breach [116]. This data was released ‘for research purposes’: the leak
led to complaints being filed with the FTC, following which the company’s
CTO resigned, and the firm fired both the employee who released the data
and the employee’s supervisor.
Another example is in movie privacy. The DVD rental firm Netflix ships
over a million DVDs a day to over 6 million U.S. customers, has a rating
system to match films to customers, and published the viewer ratings of
500,000 subscribers with their names removed. (They offered a $1m prize
for a better recommender algorithm.) In November 2007, Arvind Narayanan
and Vitaly Shmatikov showed that many subscribers could be reidentified
by comparing the anonymous records with preferences publicly expressed in
the Internet Movie Database [928]. This is partly due to the ‘long tail’ effect:
once you disregard the 100 or so movies everyone watches, people’s viewing
preferences are pretty unique. Anyway, U.S. law protects movie rental privacy,
and the attack was a serious embarrassment for Netflix.
3

Declaration of interest: I was a Special Adviser to the Committee.

295

296

Chapter 9

■

Multilateral Security

So it is important to understand what can, and what cannot, be achieved
with this technology.

9.3.2

Other Applications of Inference Control

The inference control problem was first seriously studied in the context of
census data. A census collects a vast amount of sensitive information about
individuals, then makes statistical summaries of it available by geographical
(and governmental) units such as regions, districts and wards. This information
is used to determine electoral districts, to set levels of government funding for
public services, and as inputs to all sorts of other policy decisions. The census
problem is somewhat simpler than the medical record problem as the data are
rather restricted and in a standard format (age, sex, race, income, number of
children, highest educational attainment, and so on).
There are two broad approaches, depending on whether the data are deidentified before or during processing — or equivalently whether the software
that will process the data is untrusted or trusted.
An example of the first kind of processing comes from the treatment of
U.S. census data until the 1960’s. The procedure then was that one record
in a thousand was made available on tape — minus names, exact addresses
and other sensitive data. There was also noise added to the data in order
to prevent people with some extra knowledge (such as of the salaries paid
by the employer in a company town) from tracing individuals. In addition
to the sample records, local averages were also given for people selected
by various attributes. But records with extreme values — such as very high
incomes — were suppressed. The reason for this is that a wealthy family living
in a small village might make a significant difference to the per-capita village
income. So their income might be deduced by comparing the village’s average
income with that of other villages nearby.
In the second type of processing, identifiable data are retained in a database,
and privacy protection comes from controlling the kind of queries that may
be made. Early attempts at this were not very successful, and various attacks
were proposed on the processing used at that time by the U.S. census. The
question was whether it was possible to construct a number of enquiries about
samples containing a target individual, and work back to obtain supposedly
confidential information about that individual.
If our census system allows a wide range of statistical queries, such as ‘tell
me the number of households headed by a man earning between $50,000
and $55,000’, ‘tell me the proportion of households headed by a man aged
40–45 years earning between $50,000 and $55,000’, ‘tell me the proportion
of households headed by a man earning between $50,000 and $55,000 whose
children have grown up and left home’, and so on, then an attacker can
quickly home in on an individual. Such queries, in which we add additional

9.3 Inference Control

circumstantial information in order to defeat averaging and other controls, are
known as trackers. They are usually easy to construct.
A problem related to inference is that an opponent who gets hold of a
number of unclassified files might deduce sensitive information from them.
For example, a New Zealand journalist deduced the identities of many officers
in GCSB (that country’s equivalent of the NSA) by examining lists of service
personnel and looking for patterns of postings over time [576]. Intelligence
officers’ cover postings might also be blown if an opponent gets hold of the
internal phone book for the unit where the officer is supposed to be posted,
and doesn’t find his name there. The army list might be public, and the phone
book ‘Restricted’; but the fact that a given officer is involved in intelligence
work might be ‘Secret’. Combining low level sources to draw a high level
conclusion is known as an aggregation attack. It is related to the increased risk to
personal information that arises when databases are aggregated together, thus
making more context available to the attacker and making tracker and other
attacks easier. The techniques that can be used to counter aggregation threats
are similar to those used for general inference attacks on databases, although
there are some particularly difficult problems where we have a multilevel
security policy and the inference or aggregation threats have the potential to
subvert it.

9.3.3 The Theory of Inference Control
A theory of inference control was developed by Denning and others in late
1970s and early 1980s, largely in response to problems of census bureaux [369].
The developers of many modern privacy systems are often unaware of this
work, and repeat many of the mistakes of the 1960s. (Inference control is not
the only problem in computer security where this happens.) The following is
an overview of the most important ideas.
A characteristic formula is the expression (in some database query language)
that selects a set, known as the query set, of records. An example might be ‘all
female employees of the Computer Laboratory at the grade of professor’. The
smallest query sets, obtained by the logical AND of all the attributes (or their
negations) are known as elementary sets or cells. The statistics corresponding
to query sets may be sensitive statistics if they meet criteria which I’ll discuss
below (such as the set size being too small). The objective of inference control
is to prevent the disclosure of sensitive statistics.
If we let D be the set of statistics that are disclosed and P the set which
are sensitive and must be protected, then we need D ⊆ P for privacy, where
P is the complement of P. If D = P then the protection is said to be precise.
Protection which is not precise will usually carry some cost in terms of the
range of queries which the database can answer and may thus degrade its
usefulness to its owner.

297

298

Chapter 9

9.3.3.1

■

Multilateral Security

Query Set Size Control

The simplest protection mechanism is to specify a minimum query size. As
I mentioned, New Zealand’s National Health Information System databases
will reject statistical queries whose answers would be based on fewer than six
patients’ records. But this is not enough in itself. An obvious tracker attack is
to make an enquiry on six patients’ records, and then on those records plus
the target’s. Rather than reduce the effectiveness of the database by building
in more restrictive query controls, the designers of this system opted to restrict
access to a small number of specially cleared medical statisticians.
Even so, one extra control is needed, and is often forgotten. You must
prevent the attacker from querying all but one of the records in the database.
In general, if there are N records, query set size control with a threshold of t
means that between t and N − t of them must be the subject of a query for it to
be allowed.

9.3.3.2

Trackers

Probably the most important attacks on statistical databases come from trackers. There are many simple examples. In our laboratory, only one of the full
professors is female. So we can find out her salary with just two queries:
‘Average salary professors?’ and ‘Average salary male professors?’.
This is an example of an individual tracker, a custom formula that allows
us to calculate the answer to a forbidden query indirectly. There are also
general trackers — sets of formulae which will enable any sensitive statistic to
be revealed. A somewhat depressing discovery made in the late 1970s was
that general trackers are usually easy to find. Provided the minimum query
set size n is less than a quarter of the total number of statistics N, and there
are no further restrictions on the type of queries that are allowed, then we
can find formulae that provide general trackers [372]. So tracker attacks are
easy, unless we place severe restrictions on the query set size or control the
allowed queries in some other way. (In fact results like this caused the research
community to largely lose interest in inference security as being ‘too hard’,
and this is one of the reasons that many system designers are not aware of the
problems and build databases vulnerable to trackers and other attacks.)

9.3.3.3

More Sophisticated Query Controls

There are a number of alternatives to simple query set size control. The U.S.
census, for example, uses the ‘n-respondent, k%-dominance rule’: it will not
release a statistic of which k% or more is contributed by n values or less. Other
techniques include, as I mentioned, suppressing data with extreme values. A
census bureau may deal with high-net-worth individuals in national statistics

9.3 Inference Control

but not in the local figures, while some medical databases do the same for less
common diseases. For example, a UK prescribing statistics system suppresses
sales of the AIDS drug AZT from local statistics [847]. When it was designed
in the late 1990s, there were counties with only one single patient receiving
this drug.

9.3.3.4

Cell Suppression

The next question is how to deal with the side-effects of suppressing certain
statistics. UK rules, for example, require that it be ‘unlikely that any statistical
unit, having identified themselves, could use that knowledge, by deduction,
to identify other statistical units in National Statistics outputs’ [953]. To make
this concrete, suppose that a university wants to release average marks for
various combinations of courses, so that people can check that the marking
is fair across courses. Suppose now that the table in Figure 9.4 contains the
number of students studying two science subjects, one as their major subject
and one as their minor subject.
The UK rules imply that our minimum query set size is 3 (if we set it at 2,
then either of the two students who studied ‘geology-with-chemistry’ could
trivially work out the other’s mark). Then we cannot release the average mark
for ‘geology-with-chemistry’. But if the average mark for chemistry is known,
then this mark can easily be reconstructed from the averages for ‘biologywith-chemistry’ and ‘physics-with-chemistry’. So we have to suppress at
least one other mark in the chemistry row, and for similar reasons we need to
suppress one in the geology column. But if we suppress ‘geology-with-biology’
and ‘physics-with-chemistry’, then we’d also better suppress ‘physics-withbiology’ to prevent these values being worked out in turn. Our table will now
look like Figure 9.5.
This process is called complementary cell suppression. If there are further
attributes in the database schema — for example, if figures are also broken down by race and sex, to show compliance with anti-discrimination
laws — then even more information may be lost. Where a database scheme
contains m-tuples, blanking a single cell generally means suppressing 2m − 1
Major:
Minor:
Biology
Physics
Chemistry
Geology

Biology

Physics

Chemistry

Geology

–
7
33
9

16
–
41
13

17
32
–
6

11
18
2
–

Figure 9.4: Table containing data before cell suppression

299

300

Chapter 9

■

Major:
Minor:
Biology
Physics
Chemistry
Geology

Multilateral Security

Biology

Physics

Chemistry

Geology

–
7
33
9

blanked
–
blanked
13

17
32
–
6

blanked
18
blanked
–

Figure 9.5: Table after cell suppression

other cells, arranged in a hypercube with the sensitive statistic at one vertex.
So even precise protection can rapidly make the database unusable.
Sometimes complementary cell suppression can be avoided, as when large
incomes (or rare diseases) are tabulated nationally and excluded from local
figures. But it is often necessary when we are publishing microstatistics, as in
the above tables of exam marks. Where the database is open for online queries,
we can get much of the same effect by implied queries control: we allow a query
on m attribute values only if all of the 2m implied query sets given by setting
the m attributes to true or false, have at least k records.

9.3.3.5

Maximum Order Control and the Lattice Model

The next thing we might try in order to make it harder to construct trackers
is to limit the type of inquiries that can be made. Maximum order control limits
the number of attributes that any query can have. However, to be effective, the
limit may have to be severe. One study found that of 1000 medical records,
three attributes were safe while with four attributes, one individual record
could be found and with 10 attributes most records could be isolated. A more
thorough approach (where it is feasible) is to reject queries that would partition
the sample population into too many sets.
We saw how lattices can be used in compartmented security to define a
partial order to control permitted information flows between compartments
with combinations of codewords. They can also be used in a slightly different
way to systematize query controls in some databases. If we have, for example,
three attributes A, B and C (say area of residence, birth year and medical
condition), we may find that while enquiries on any one of these attributes are
non-sensitive, as are enquiries on A and B and on B and C, the combination of
A and C might be sensitive. It follows that an enquiry on all three would not be
permissible either. So the lattice divides naturally into a ‘top half’ of prohibited
queries and a ‘bottom half’ of allowable queries, as shown in Figure 9.6.

9.3.3.6

Audit Based Control

As mentioned, some systems try to get round the limits imposed by static query
control by keeping track of who accessed what. Known as query overlap control,

9.3 Inference Control

(A, B, C)
Prohibited
(A, B)

(B, C)

(C, A)

A

B

C

Allowable

U
Figure 9.6: Table lattice for a database with three attributes

this involves rejecting any query from a user which, combined with what
the user knows already, would disclose a sensitive statistic. This may sound
perfect in theory but in practice it suffers from two usually unsurmountable
drawbacks. First, the complexity of the processing involved increases over
time, and often exponentially. Second, it’s extremely hard to be sure that your
users aren’t in collusion, or that one user has registered under two different
names. Even if your users are all honest and distinct persons today, it’s always
possible that one of them will take over another, or get taken over by a
predator, tomorrow.

9.3.3.7

Randomization

Our cell suppression example shows that if various kinds of query control are
the only protection mechanisms used in a statistical database, they can often
have an unacceptable performance penalty. So query control is often used in
conjunction with various kinds of randomization, designed to degrade the
signal-to-noise ratio from the attacker’s point of view while impairing that of
the legitimate user as little as possible.
The simplest such technique is perturbation, or adding noise with zero mean
and a known variance to the data. One way of doing this is to round or
truncate the data by some deterministic rule; another is to swap some records.
Perturbation is often not as effective as one would like, as it will tend to
damage the legitimate user’s results precisely when the sample set sizes are
small, and leave them intact when the sample sets are large (where we might
have been able to use simple query controls anyway). There is also the worry
that suitable averaging techniques might be used to eliminate some of the
added noise. A modern, sophisticated variant on the same theme is controlled
tabular adjustment where you identify the sensitive cells and replace their

301

302

Chapter 9

■

Multilateral Security

values with ‘safe’ (sufficiently different) ones, then adjust other values in the
table to restore additive relationships [330].
Often a good randomization technique is to use random sample queries. This
is another of the methods used by census bureaux. The idea is that we make
all the query sets the same size, selecting them at random from the available
relevant statistics. Thus all the released data are computed from small samples
rather than from the whole database. If this random selection is done using a
pseudorandom number generator keyed to the input query, then the results
will have the virtue of repeatability. Random sample queries are a natural
protection mechanism for large medical databases, where the correlations
being investigated are often such that a sample of a few hundred is sufficient.
For example, when investigating the correlation between a given disease and
some aspect of lifestyle, the correlation must be strong before doctors will
advise patients to make radical changes to their way of life, or take other
actions that might have undesirable side effects. If a teaching hospital has
records on five million patients, and five thousand have the disease being
investigated, then a randomly selected sample of two hundred sufferers might
be all the researcher could use.
This doesn’t work so well where the disease is rare, or where for other
reasons there is only a small number of relevant statistics. A possible strategy
here is randomized response, where we randomly restrict the data we collect (the
subjects’ responses). For example, if the three variables under investigation
are obesity, smoking and AIDS, we might ask each subject with HIV infection
to record whether they smoke or whether they are overweight, but not both.
Of course, this can limit the value of the data.

9.3.4

Limitations of Generic Approaches

As with any protection technology, statistical security can only be evaluated
in a particular environment and against a particular threat model. Whether it
is adequate or not depends to an even greater extent than usual on the details
of the application.
An instructive example is a system used for analyzing trends in drug
prescribing. Here, prescriptions are collected (minus patient names) from
pharmacies. A further stage of de-identification removes the doctors’ identities,
and the information is then sold to drug company marketing departments.
The system has to protect the privacy of doctors as well as of patients: the last
thing a busy family doctor wants is to be pestered by a drug rep for prescribing
a competitor’s brands.
One early prototype of this system merely replaced the names of doctors
in a cell of four or five practices with ‘doctor A’, ‘doctor B’ and so on, as
in Figure 9.7. We realised that an alert drug rep could identify doctors from
prescribing patterns, by noticing, for example, ‘‘Well, doctor B must be Susan

9.3 Inference Control

Week:
Doctor A
Doctor B
Doctor C
Doctor D

1
17
25
32
16

2
26
31
30
19

3
19
9
39
18

4
22
29
27
13

Figure 9.7: Sample of de-identified drug prescribing data

Jones because she went skiing in the third week in January and look at the
fall-off in prescriptions here. And doctor C is probably her partner Mervyn
Smith who’ll have been covering for her’’ The fix was to replace absolute
numbers of prescriptions with the percentage of each doctor’s prescribing
which went on each particular drug, to drop some doctors at random, and to
randomly perturb the timing by shifting the figures backwards or forwards a
few weeks [847].
This is a good example of the sort of system where the inference control problem can have a robust solution. The application is well-defined, the database
is not too rich, and the allowable queries are fairly simple. Indeed, this system
was the subject of litigation; the UK government’s Department of Health sued
the database operator, alleging that the database might compromise privacy.
Their motive was to maintain a monopoly on the supply of such information
to industry. They lost, and this established the precedent that (in Britain at
least) inference security controls may, if they are robust, exempt statistical data
from being considered as ‘personal information’ for the purpose of privacy
laws [1204].
In general, though, it’s not so easy. For a start, de-identification doesn’t
compose: it’s easy to have two separate applications, each of which provides
the same results via anonymized versions of the same data, but where an
attacker with access to both of them can easily identify individuals. In the
general case, contextual knowledge is extremely hard to quantify, and is likely
to grow over time. Latanya Sweeney has shown that even the HCFA’s ‘publicuse’ files can often be reidentified by cross-correlating them with commercial
databases [1235]: for example, most U.S. citizens can be identified by their
ZIP code plus their gender and date of birth. Such data detective work is an
important part of assessing the level of protection which an actual statistical
database gives, just as we only have confidence in cryptographic algorithms
which have withstood extensive analysis by capable motivated opponents.
The emergence of social networks since 2004 has made inference control much
harder wherever they can be brought to bear; I will discuss this when we get
to social networks in section 23.3.3. And even without cross-correlation, there
may be contextual information available internally. Users of medical research
databases are often doctors who have normal access to parts of the patient
record databases from which the statistical data are drawn.

303

304

Chapter 9

9.3.4.1

■

Multilateral Security

Active Attacks

Active attacks are particularly powerful. These are where users have the ability
to insert or delete records into the database. A user might add records to create
a group that contains the target’s record plus those of a number of nonexistent
subjects created by himself. One (imperfect) countermeasure is add or delete
new records in batches. Taking this to an extreme gives partitioning — in which
records are added in groups and any query must be answered with respect
to all of them or none. However, this is once more equivalent to publishing
tables of microstatistics.
Active attacks are not limited to data, but can also target metadata. A nice
example, due to Whit Diffie, is the chosen drug attack. Suppose a drug company
has access through a statistical system to the amounts of money spent on
behalf of various groups of patients and wishes to find out which patients
are receiving which drug, in order to direct its marketing better (there was a
scandal in Quebec about just such an inference attack). A possible trick is to set
the drug prices in such a way as to make the resulting equations easy to solve.
A prominent case at the turn of the century was a medical research database
in Iceland. The plan was for three linked databases: one with the nation’s
medical records, a second with the genealogy of the whole population, and a
third with genetic data acquired from sequencing. The rationale was that since
Iceland’s population is largely descended from a few founding families who
settled there about a thousand years ago, there is much less genic variance than
in the general human population and so genes for hereditary illnesses should
be much easier to find. A Swiss drug company bankrolled the construction
of the database, and the Reykjavik government embraced it as a means
of modernising the country’s health IT infrastructure and simultaneously
creating a few hundred high-tech jobs in medical research. Iceland’s doctors,
however, mostly reacted negatively, seeing the system as a threat both to
patient privacy and professional autonomy.
The privacy problem in the Icelandic database was more acute than in the
general case. For example, by linking medical records to genealogies, which
are in any case public (genealogy is a common Icelandic hobby), patients can
be identified by such factors as the number of their uncles, aunts, great-uncles,
great-aunts and so on — in effect by the shape of their family trees. There
was much debate about whether the design could even theoretically meet
legal privacy requirements [47], and European privacy officials expressed
grave concern about the possible consequences for Europe’s system of privacy
laws [349]. The Icelandic government pressed ahead with it anyway, with a
patient opt-out. Many doctors advised patients to opt out, and 11% of the population did so. Eventually, the Icelandic Supreme Court found that European
privacy law required the database to be opt-in rather than opt-out. In addition,
many Icelanders had invested in the database company, and lost money when

9.3 Inference Control

its share value sank at the end of the dotcom boom. Nowadays about half the
population have opted in to the system and the controversy is defused.
My own view, for what it’s worth, is that patient consent is the key to
effective medical research. This not only allows full access to data, without
the problems we’ve been discussing in this section, but provides motivated
subjects and much higher-quality clinical information than can be harvested
simply as a byproduct of normal clinical activities. For example, a network
of researchers into ALS (the motor-neurone disease from which Cambridge
astronomer Stephen Hawking suffers) shares fully-identifiable information
between doctors and other researchers in over a dozen countries with the full
consent of the patients and their families. This network allows data sharing
between Germany, with very strong privacy laws, and Japan, with almost
none; and data continued to be shared between researchers in the USA and
Serbia even when the USAF was bombing Serbia. The consent model is
spreading. Britain’s biggest medical charity is funding a ‘Biobank’ database in
which several hundred thousand volunteers will be asked to give researchers
not just answers to an extensive questionnaire and full access to their records
for the rest of their lives, but also to lodge blood samples so that those who
develop interesting diseases in later life can have their genetic and proteomic
makeup analysed.

9.3.5 The Value of Imperfect Protection
So doing de-identification right is hard, and the issues can be politically
fraught. The best way to solve the inference control problem is to avoid it,
for example by recruiting volunteers for your medical research rather than
recycling data collected for other purposes. But there are applications where
it’s used, and applications where it’s all that’s available. An example was
the epidemic of HIV/AIDS; in the 1980s and 1990s researchers struggling to
understand what was going on had little choice but to use medical data that
had been originally collected for other purposes. Another example, of course,
is the census. In such applications the protection you can provide will be
imperfect. How do you cope with that?
Some kinds of security mechanism may be worse than useless if they can be
compromised. Weak encryption is a good example. The main problem facing
the world’s signals intelligence agencies is traffic selection — how to filter out
interesting nuggets from the mass of international phone, fax, email and other
traffic. A terrorist who helpfully encrypts his important traffic does this part
of the police’s job for them. If the encryption algorithm used is breakable, or if
the end systems can be hacked, then the net result is worse than if the traffic
had been sent in clear.
Statistical security is not generally like this. The main threat to databases of
personal information is often mission creep. Once an organization has access to

305

306

Chapter 9

■

Multilateral Security

potentially valuable data, then all sorts of ways of exploiting that value will be
developed. Some of these are likely to be highly objectionable; one topical U.S.
example is the resale of medical records to banks for use in filtering loan applications. However, even an imperfect de-identification system may destroy the
value of medical data to a bank’s loan department. If only five percent of
the patients can be identified, and then only with effort, then the bank may
decide that it’s simpler to tell loan applicants to take out their own insurance
and let the insurance companies send out medical questionnaires if they wish.
So de-identification can help prevent mission creep, even if the main effect is
prophylaxis against future harm rather than treatment of existing defects.
As well as harming privacy, mission creep can have safety implications. In
the UK, diabetic registers were set up in the 1990s to monitor the quality of
diabetes care; they were databases to which GPs, hospital consultants, nurses
and opthalmologists could upload test results, so that important indicators
would not be missed. As hospitals had no working email system, they were
promptly abused to provide a rudimentary messaging system between hospitals and general practice. But as the diabetes registers were never designed as
communications systems, they lacked the safety and other mechanisms that
such systems should have had if they were to be used for clinical data. Even
rudimentary de-identification would have prevented this abuse and motivated
diabetologists to get email working instead.
So in statistical security, the question of whether one should let the best be
the enemy of the good can require a finer judgment call than elsewhere.

9.4

The Residual Problem

The above two sections may have convinced you that the problem of managing
medical record privacy in the context of immediate care (such as in a hospital)
is reasonably straightforward, while in the context of secondary databases
(such as for research, audit and cost control) there are statistical security
techniques which, with care, can solve much of the problem. Somewhat similar
techniques can be used to manage highly sensitive commercial data such as
details of forthcoming mergers and acquisitions in an investment bank, and
even intelligence information. (There was a lot of interest in the BMA model
from people designing police intelligence systems.) In all cases, the underlying
concept is that the really secret material is restricted to a compartment of a
small number of identified individuals, and less secret versions of the data may
be manufactured for wider use. This involves not just suppressing the names
of the patients, or spies, or target companies, but also careful management of
contextual and other information by which they might be re-identified.
But making such systems work well in real life is much harder than it looks.
First, determining the sensitivity level of information is fiendishly difficult,

9.4 The Residual Problem

and many initial expectations turn out to be wrong. You might expect, for
example, that HIV status would be the most sensitive medical data there is;
yet many HIV sufferers are quite open about their status. You might also
expect that people would rather entrust sensitive personal health information
to a healthcare professional such as a doctor or pharmacist rather than to a
marketing database. Yet many women are so sensitive about the purchase
of feminine hygiene products that, rather than going into a pharmacy and
buying them for cash, they prefer to use an automatic checkout facility in a
supermarket — even if this means they have to use their store card and credit
card, so that the purchase is linked to their name and stays on the marketing
database forever. The actual embarrassment of being seen with a packet of
tampons is immediate, and outweighs the future embarrassment of being sent
discount coupons for baby wear six months after the menopause.
Second, it is extraordinarily difficult to exclude single points of failure, no
matter how hard you try to build watertight compartments. The CIA’s Soviet
assets were compromised by Aldritch Ames — who as a senior counterintelligence man had access to too many compartments. The KGB’s overseas
operations were similarly compromised by Vassily Mitrokhin — an officer
who’d become disillusioned with communism after 1968 and who was sent to
work in the archives while waiting for his pension [77]. And in March 2007, historians Margo Anderson and William Seltzer found, that contrary to decades
of denials, census data was used in 1943 to round up Japanese-Americans
for internment [1142]. The single point of failure there appears to have been
Census Bureau director JC Capt, who unlawfully released the data to the
Secret Service following a request from Treasury Secretary HC Morgenthau.
The Bureau has since publicly apologised [893].
In medicine, many of the hard problems lie in the systems that process
medical claims for payment. When a patient is treated and a request for
payment sent to the insurer, it has not just full details of the illness, the
treatment and the cost, but also the patient’s name, insurance number and
other details such as date of birth. There have been proposals for payment to
be effected using anonymous credit cards [191], but as far as I am aware none
of them has been fielded. Insurers want to know which patients, and which
doctors, are the most expensive. In fact, during a debate on medical privacy at
an IEEE conference in 1996 — just as HIPAA was being pushed through the
U.S. Congress — a representative of a large systems house declared that the
medical records of 8 million Americans were one of his company’s strategic
assets, which they would never give up. This holds whether the insurer is
a private insurance company (or employer) or a government-owned health
authority, such as HCFA, the VA, or Britain’s National Health Service. Once an
insurer possesses large quantities of personal health information, it becomes
very reluctant to delete it. Its potential future value, in all sorts of applications

307

308

Chapter 9

■

Multilateral Security

from cost control through research to marketing, is immediate and obvious,
while patients’ privacy concerns are not.
In the USA, the retention of copies of medical records by insurers, employers and others is widely seen as a serious problem. Writers from such widely
different political viewpoints as the communitarian Amitai Etzioni [441] and
the libertarian Simson Garfinkel [515] agree on this point, if on little else.
As mentioned, HIPAA only empowered the DHHS to regulate health plans,
healthcare clearinghouses, and healthcare providers, leaving many organizations that process medical data (such as lawyers, employers and universities)
outside its scope. In fact, Microsoft’s recent announcement that it would set
up a ‘HealthVault’ to guard your medical records was met with a sharp retort
from privacy activists that since Microsoft isn’t a ‘covered entity’ as specified
by HIPAA, putting your medical data there would place it outside HIPAA’s
protection [81].
What lessons can be drawn from other countries?
Medical privacy is strongly conditioned by how people pay for healthcare.
In Britain, the government pays for most healthcare, and the attempts of
successive British governments to centralise medical records for cost control
and management purposes have led to over a decade of conflict with doctors
and with patients’ associations. In Germany, the richer people use private
insurers (who are bound by tight data protection laws), while the poor use
state health insurers that are run by doctors, so non-doctors don’t have access
to records. Singapore residents pay into compulsory savings accounts from
their wages and use them to pay for healthcare; the government steps in to
insure expensive procedures, but most doctor visits are paid by the patient
directly. Patients who stay healthy and accumulate a surplus can add some of
it to their pension and pass the rest to their heirs. The most radical solution is in
Japan, where costs are controlled by regulating fees: doctors are discouraged
from performing expensive procedures such as heart transplants by pricing
them below cost. In the mid-1990s, healthcare took up some 3% of GNP in
Japan, versus 7–9% for the typical developed country and 15% for America;
since then the figures have risen by a percent or so, but the general rankings
remain the same. Japanese (and Singaporeans) pay less for healthcare than
Europeans, and Americans pay more. The curious thing is that Japanese (and
Singaporeans) live longer than Europeans, who live longer than Americans.
Life expectancy and medical costs seem to be negatively correlated.
To sum up, the problem of health record privacy is not just a socio-technical
one but socio-technico-political. Whether large quantities of medical records
accumulate in one database depends on how the health care system is organized, and whether these are destroyed — or de-identified — after payment
has been processed is more to do with institutional structures, incentives and
regulation than technology. In such debates, one role of the security engineer
is to get policymakers to understand the likely consequences of their actions.

9.5 Summary

Privacy is poorest in countries that fail to align incentives properly, and as
a result have detailed cost oversight of individual treatments — whether by
insurers / employers, as in the USA, or by bureaucrats as in Britain.
In the UK, a scandal broke in November 2007 when the tax authorities
lost the records of 25 million people. The records of all the nation’s children
and their families — including names, addresses, phone numbers and tha
parents’ bank account details — were burned on two CDs for dispatch to
the National Audit Office, and lost in the post. The Prime Minister had to
apologise to Parliament and promised to make good any resulting ‘identify
theft’ losses. In the aftermath, there has been wide public questioning of his
government’s programme to build ever-larger central databases of citizens’
personal information — not just for taxation but for medical research, healthservice administration, and child welfare. As I write in December 2007, the
feeling in London is that plans for a national ID card are effectively dead, as is a
proposal to build a database of all vehicle movements to facilitate road pricing.
The National Health Service is continuing to build central health databases
against growing medical resistance, but the opposition Conservative Party
(which now has a clear lead in the polls) have promised to abolish not just the
ID card system but proposed children’s databases if they win the next election.
Other privacy problems also tend to have a serious political entanglement.
Bank customer privacy can be tied up with the bank’s internal politics; the
strongest driver for privacy protection may come from branch managers’
reluctance to let other branches learn about their customers. Access to criminal
records and intelligence depends on how law enforcement agencies decide to
share data with each other, and the choices they make internally about whether
access to highly sensitive information about sources and methods should
be decentralized (risking occasional losses), or centralized (bringing lowerprobability but higher-cost exposure to a traitor at head office). The world
since 9/11 has moved sharply towards centralisation; expect a high-profile
traitor like Aldrich Ames to come along sometime soon.

9.5

Summary

In this chapter, we looked at the problem of assuring the privacy of medical
records. This is typical of a number of information security problems, ranging
from the protection of national intelligence data through professional practice
in general to the protection of census data.
It turns out that with medical records there is an easy problem, a harder
problem, and a really hard problem.
The easy problem is setting up systems of access controls so that access to
a particular record is limited to a sensible number of staff. Such systems can
be designed largely by automating existing working practices, and role-based

309

310

Chapter 9

■

Multilateral Security

access controls are currently the technology of choice. The harder problem
is statistical security — how one designs databases of medical records (or
census returns) so as to allow researchers to make statistical enquiries without
compromising individuals’ privacy. The hardest problem is how to manage
the interface between the two, and in the specific case of medicine, how to
prevent the spread of payment information. The only realistic solution for this
lies in regulation.
Medical systems also teach us about the limits of some privacy enhancing technologies, such as de-identification. While making medical records
anonymous in research databases can help mitigate the consequences of unauthorised access and prevent mission creep, it’s by no means bulletproof. Rich
data about real people can usually be re-identified. The mechanisms used in
healthcare to deal with this problem are worth studying.

Research Problems
In the near future, a lot of medical treatment may involve genetic information.
So your medical records may involve personal health information about your
parents, siblings, cousins and so on. How can privacy models be extended
to deal with multiple individuals? For example, in many countries you have
the right not to know the outcome of a DNA test that a relative has for an
inheritable disease such as Huntington’s Chorea, as it may affect the odds that
you have the disease too. Your relative does have a right to know, and may tell
others. This is a problem not just for technology, but also for privacy law [1231]
Are there any ways of linking together access control policies for privacy
with statistical security? Can there be such a thing as seamless privacy
where everything fits neatly together? Or would you end up giving patients
an extremely complex set of access control options — like Facebook’s but
worse — in which each patient had to wade through dozens of pages of
options and approve or deny permission for her data to be used in each of
dozens of secondary applications and research projects? In short, are there any
useful and useable abstractions?
What other ways of writing privacy policies are there? For example, are
there useful ways to combine BMA and Chinese Wall? Are there any ways,
whether technical or economic, of aligning the data subject’s interest with
those of the system operator and other stakeholders?

Further Reading
The literature on compartmented-mode security is somewhat scattered: most
of the public domain papers are in the proceedings of the NCSC/NISSC and

Further Reading

ACSAC conferences cited in detail at the end of Chapter 8. Standard textbooks
such as Amoroso [27] and Gollmann [537] cover the basics of the lattice and
Chinese Wall models.
For the BMA model see the policy document itself — the Blue Book [37],
the shorter version at [38], and the proceedings of the conference on the
policy [43]. See also the papers on the pilot system at Hastings [366, 367]. For
more on Japanese healthcare, see [263]. For a National Research Council study
of medical privacy issues in the USA, see [951]; there is also an HHS report on
the use of de-identified data in research at [816].
As for inference control, this has become an active research field again in the
last few years, with regular conferences on ‘Privacy in Statistical Databases’;
see the proceedings of these events to catch up with current frontiers. Denning’s book [369] is the classic reference, and still worth a look; there’s an
update at [374]. A more modern textbook on database security is the one
by Castano et al [276]. The most comprehensive resource, though, from the
practical viewpoint — with links to a vast range of practical literature across a
number of application areas — may be the website of the American Statistical
Association [26]. The standard reference for people involved in government
work is the Federal Committee on Statistical Methodology’s ‘Report on Statistical Disclosure Limitation Methodology’ which provides a good introduction to
the standard tools and describes the methods used in various U.S. departments
and agencies [455]. As an example of a quite different application, Mark Allman and Vern Paxson discuss the problems of anonymizing IP packet traces
for network systems research in [23].
Finally, Margo Anderson and William Seltzer’s papers on the abuses of
census data in the USA, particularly during World War 2, can be found at [31].

311

CHAPTER

10
Banking and Bookkeeping
The arguments of lawyers and engineers pass through one
another like angry ghosts.
— Nick Bohm, Brian Gladman and Ian Brown [201]
Computers are not (yet?) capable of being reasonable
any more than is a Second Lieutenant.
— Casey Schaufler
Against stupidity, the Gods themselves contend in vain.
— JC Friedrich von Schiller

10.1

Introduction

Banking systems range from cash machine networks and credit card
processing, both online and offline, through high-value interbank money
transfer systems, to the back-end bookkeeping systems that keep track of it all
and settle up afterwards. There are specialised systems for everything from
stock trading to bills of lading; and large companies have internal bookkeeping and cash management systems that duplicate many of the functions of
a bank.
Such systems are important for a number of reasons. First, an understanding of transaction processing is a prerequisite for tackling the broader
problems of electronic commerce and fraud. Many dotcom firms fell down
badly on elementary bookkeeping; in the rush to raise money and build
web sites, traditional business discipline was ignored. The collapse of Enron
led to stiffened board-level accountability for internal control; laws such as
313

314

Chapter 10

■

Banking and Bookkeeping

Sarbanes-Oxley and Gramm-Leach-Bliley now drive much of the investment
in information security. When you propose protection mechanisms to a client,
one of the first things you’re likely to be asked is the extent to which they’ll
help directors of the company discharge their fiduciary responsibilities to
shareholders.
Second, bookkeeping was for many years the mainstay of the computer
industry, with banking its most intensive application area. Personal applications such as web browsers and Office might now run on more machines,
but accounting is still the critical application for the average business. So the
protection of bookkeeping systems is of great practical importance. It also
gives us a well-understood model of protection in which confidentiality plays
little role, but where the integrity of records (and their immutability once
made) is of paramount importance.
Third, transaction processing systems — whether for small debits such as
$50 cash machine withdrawals, or multimillion dollar wire transfers — were
the application that launched commercial cryptology. Banking applications
drove the development not just of encryption algorithms and protocols, but
also of the supporting technology such as tamper-resistant cryptographic
processors. These processors provide an important and interesting example of
a trusted computing base that is quite different from the hardened operating
systems discussed in the context of multilevel security. Many instructive
mistakes were first made (or at least publicly documented) in the area of
commercial cryptography. The problem of how to interface crypto with access
control was studied by financial cryptographers before any others in the open
research community.
Finally, banking systems provide another example of multilateral security — but aimed at authenticity rather than confidentiality. A banking system
should prevent customers from cheating each other, or the bank; it should
prevent bank employees from cheating the bank, or its customers; and the
evidence it provides should be sufficiently strong that none of these principals
can get away with falsely accusing another principal of cheating.
In this chapter, I’ll first describe the bookkeeping systems used to keep
track of assets despite occasional corrupt staff; these are fairly typical of
accounting systems used by other companies too. I’ll then describe the banks’
principal international funds-transfer systems; similar systems are used to
settle securities transactions and to manage trade documents such as bills of
lading. Next, I’ll describe ATM systems, which are increasingly the public face
of banking, and whose technology has been adopted in applications such as
utility meters; and then I’ll tell the story of credit cards, which have become the
main payment mechanism online. I’ll then move on to more recent technical
advances, including the smartcards recently introduced in Europe, RFID credit

10.1 Introduction

cards, and nonbank payment services such as PayPal. I’ll wrap up with some
points on money laundering, and what controls really work against fraud.

10.1.1 The Origins of Bookkeeping
Bookkeeping appears to have started in the Neolithic Middle East in about
8500 BC, just after the invention of agriculture [1122]. When people started
to produce surplus food, they started to store and trade it. Suddenly they
needed a way to keep track of which villager had put how much in the
communal warehouse. To start with, each unit of food (sheep, wheat, oil, . . .)
was represented by a clay token, or bulla, which was placed inside a clay
envelope and sealed by rolling it with the pattern of the warehouse keeper.
(See Figure 10.1.) When the farmer wanted to get his food back, the seal was
broken by the keeper in the presence of a witness. (This may be the oldest
known security protocol.) By about 3000BC, this had led to the invention of
writing [1018]; after another thousand years, we find equivalents of promissory
notes, bills of lading, and so on. At about the same time, metal ingots started
to be used as an intermediate commodity, often sealed inside a bulla by an
assayer. In 700BC, Lydia’s King Croesus started stamping the metal directly
and thus invented coins [1045]; by the Athens of Pericles, there were a number
of wealthy individuals in business as bankers [531].

Figure 10.1: Clay envelope and its content of tokens representing 7 jars of oil, from Uruk,
present day Iraq, ca. 3300 BC (Courtesy Denise Schmandt-Besserat and the Louvre Museum)

315

316

Chapter 10

■

Banking and Bookkeeping

The next significant innovation dates to late medieval times. As the dark
ages came to a close and trade started to grow, some businesses became too
large for a single family to manage. The earliest of the recognisably modern
banks date to this period; by having branches in a number of cities, they could
finance trade efficiently. But as the economy grew, it was necessary to hire
managers from outside, and the owner’s family could not supervise them
closely. This brought with it an increased risk of fraud, and the mechanism
that evolved to control it was double-entry bookkeeping. People used to think this
was invented in Italy sometime in the 1300s, though the first book on it did not
appear until 1494, after the printing press came along [355]. Recently, however,
historians have found double-entry records created by Jewish merchants in
twelfth-century Cairo, and it’s now believed that the Italians learned the
technique from them [1140].

10.1.2

Double-Entry Bookkeeping

The idea behind double-entry bookkeeping is extremely simple, as with most
hugely influential ideas. Each transaction is posted to two separate books, as
a credit in one and a debit in the other. For example, when a firm sells a
customer $100 worth of goods on credit, it posts a $100 credit on the Sales
account, and a $100 debit onto the Receivables account. When the customer
pays the money, it will credit the Receivables account (thereby reducing the
asset of money receivable), and debit the Cash account. (The principle taught
in accountancy school is ‘debit the receiver, credit the giver’.) At the end of
the day, the books should balance, that is, add up to zero; the assets and the
liabilities should be equal. (If the firm has made some profit, then this is a
liability to the shareholders.) In all but the smallest firms, the books will be
kept by different clerks, and have to balance at the end of every month (at
banks, every day).
By suitable design of the ledger system, we can see to it that each shop, or
branch, can be balanced separately. Thus each cashier will balance her cash
tray before locking it in the vault overnight; the debits in the cash legder
should exactly balance the physical banknotes she’s collected. So most frauds
need the collusion of two or more members of staff; and this principle of split
responsibility, also known as dual control, is complemented by audit. Not only
are the books audited at year end, but there are random audits too; a team of
inspectors may descend on a branch at no notice and insist that all the books
are balanced before the staff go home.

10.1.3

A Telegraphic History of E-commerce

Many of the problems afflicting e-businesses stem from the popular notion
that e-commerce is something completely new, invented in the mid-1990s.
This is simply untrue.

10.2 How Bank Computer Systems Work

Various kinds of visual signalling were deployed from classical times,
including heliographs (which used mirrors to flash sunlight at the receiver),
semaphones (which used the positions of moving arms to signal letters and
numbers) and flags. Land-based systems sent messages along chains of beacon
towers, and naval systems relayed them between ships. To begin with, their
use was military, but after the Napoleonic War the French government opened
its heliograph network to commercial use. Very soon the first frauds were
carried out. For two years up till they were discovered in 1836, two bankers
bribed an operator to signal the movements of the stock market to them
covertly by making errors in transmissions that they could observe from a safe
distance. Other techniques were devised to signal the results of horseraces.
Various laws were passed to criminalise this kind of activity but they were
ineffective. The only solution for the bookies was to ‘call time’ by a clock,
rather than waiting for the result and hoping that they were the first to hear it.
From the 1760’s to the 1840’s, the electric telegraph was developed by a
number of pioneers, of whom the most influential was Samuel Morse. He
persuaded Congress in 1842 to fund an experimental line from Washington
to Baltimore; this so impressed people that serious commercial investment
started, and by the end of that decade there were 12,000 miles of line operated
by 20 companies. This was remarkably like the Internet boom of the late 1990’s.
Banks were the first big users of the telegraph, and they decided that
they needed technical protection mechanisms to prevent transactions being
altered by crooked operators en route. (I discussed the test key systems
they developed for the purpose in section 5.2.4.) Telegrams were also used
to create national markets. For the first time, commodity traders in New
York could find out within minutes what prices had been set in auctions
in Chicago, and fishing skippers arriving in Boston could find out the price
of cod in Gloucester. The history of the period shows that most of the
concepts and problems of e-commerce were familiar to the Victorians [1215].
How do you know who you’re speaking to? How do you know if they’re
trustworthy? How do you know whether the goods will be delivered, and
whether payments will arrive? The answers found in the nineteenth century
involved intermediaries — principally banks who helped business manage
risk using instruments such as references, guarantees and letters of credit.

10.2

How Bank Computer Systems Work

Banks were among the first large organizations to use computers for bookkeeping. They started in the late 1950s and early 1960s with applications such
as check processing, and once they found that even the slow and expensive
computers of that era were much cheaper than armies of clerks, they proceeded to automate most of the rest of their back-office operations during the

317

318

Chapter 10

■

Banking and Bookkeeping

1960s and 1970s. The 1960s saw banks offering automated payroll services to
their corporate customers, and by the 1970s they were supporting business-tobusiness e-commerce based on electronic data interchange (EDI), whereby firms
from General Motors to Marks and Spencer built systems that enabled them
to link up their computers to their suppliers’ so that goods could be ordered
automatically. Travel agents built similar systems to order tickets in real time
from airlines. ATMs arrived en masse in the 1970s, and online banking systems
in the 1980s; web-based banking followed in the 1990s. Yet the fancy front-end
systems still rely on traditional back-office automation for maintaining account
data and performing settlement.
Computer systems used for bookkeeping typically claim to implement
variations on the double-entry theme. But the quality of control is highly
variable. The double-entry features may be just a skin in the user interface,
while the underlying file formats have no integrity controls. And if the ledgers
are all kept on the same system, someone with root access — or with physical
access and a debugging tool — may be able to change the records so that
the balancing controls are bypassed. It may also be possible to evade the
balancing controls in various ways; staff may notice bugs in the software and
take advantage of them. Despite all these problems, the law in most developed
countries requires companies to have effective internal controls, and makes the
managers responsible for them. Such laws are the main drivers of investment
in information security mechanisms, but they also a reason for much wasted
investment. So we need to look at the mechanics of electronic bookkeeping in
a more detail.
A typical banking system has a number of data structures. There is an
account master file which contains each customer’s current balance together
with previous transactions for a period of perhaps ninety days; a number of
ledgers which track cash and other assets on their way through the system;
various journals which hold transactions that have been received from teller
stations, cash machines, check sorters and so on, but not yet entered in the
ledgers; and an audit trail that records which staff member did what and when.
The processing software that acts on these data structures will include a
suite of overnight batch processing programs, which apply the transactions
from the journals to the various ledgers and the account master file. The
online processing will include a number of modules which post transactions
to the relevant combinations of ledgers. So when a customer pays $100 into
his savings account the teller will make a transaction which records a credit to
the customer’s savings account ledger of $100 while debiting the same amount
to the cash ledger recording the amount of money in the drawer. The fact that
all the ledgers should always add up to zero provides an important check; if
the bank (or one of its branches) is ever out of balance, an alarm will go off
and people will start looking for the cause.

10.2 How Bank Computer Systems Work

The invariant provided by the ledger system is checked daily during the
overnight batch run, and means that a programmer who wants to add to his
own account balance will have to take the money from some other account,
rather than just creating it out of thin air by tweaking the account master file.
Just as in a traditional business one has different ledgers managed by different
clerks, so in a banking data processing shop there are different programmers
in charge of different subsystems. In addition, all code is subjected to scrutiny
by an internal auditor, and to testing by a separate test department. Once it
has been approved, it will be run on a production machine that does not have
a development environment, but only approved object code and data.

10.2.1 The Clark-Wilson Security Policy Model
Although such systems had been in the field since the 1960s, a formal model
of their security policy was only introduced in 1987, by Dave Clark and Dave
Wilson (the former a computer scientist, and the latter an accountant) [295]. In
their model, some data items are constrained so that they can only be acted on
by a certain set of transformation procedures.
More formally, there are special procedures whereby data can be input —
turned from an unconstrained data item, or UDI, into a constrained data item, or
CDI; integrity verification procedures (IVP’s) to check the validity of any CDI
(e.g., that the books balance); and transformation procedures (TPs) which may
be thought of in the banking case as transactions which preserve balance. In
the general formulation, they maintain the integrity of CDIs; they also write
enough information to an append-only CDI (the audit trail) for transactions to
be reconstructed. Access control is by means of triples (subject, TP, CDI), which
are so structured that a dual control policy is enforced. In the formulation
in [27]:
1. the system will have an IVP for validating the integrity of any CDI;
2. the application of a TP to any CDI must maintain its integrity;
3. a CDI can only be changed by a TP;
4. subjects can only initiate certain TPs on certain CDIs;
5. triples must enforce an appropriate separation-of-duty policy on
subjects;
6. certain special TPs on UDIs can produce CDIs as output;
7. each application of a TP must cause enough information to reconstruct it
to be written to a special append-only CDI;
8. the system must authenticate subjects attempting to initiate a TP;
9. the system must let only special subjects (i.e., security officers) make
changes to authorization-related lists.

319

320

Chapter 10

■

Banking and Bookkeeping

A number of things bear saying about Clark-Wilson.
First, unlike Bell-LaPadula, Clark-Wilson involves maintaining state. Even
disregarding the audit trail, this is usually necessary for dual control as you
have to keep track of which transactions have been partially approved — such
as those approved by only one manager when two are needed. If dual control
is implemented using access control mechanisms, it typically means holding
partially approved transactions in a special journal file. This means that some
of the user state is actually security state, which in turn makes the trusted
computing base harder to define. If it is implemented using crypto instead,
such as by having managers attach digital signatures to transactions of which
they approve, then there can be problems managing all the partially approved
transactions so that they get to a second approver in time.
Second, the model doesn’t do everything. It captures the idea that state transitions should preserve an invariant such as balance, but not that state
transitions should be correct. Incorrect transitions, such as paying into the
wrong bank account, are not prevented by this model.
Third, Clark-Wilson ducks the hardest question, namely: how do we control
the risks from dishonest staff? Rule 5 says that ‘an appropriate separation of
duty policy’ must be supported, but nothing about what this means. Indeed,
it’s very hard to find any systematic description in the accounting literature of
how you design internal controls — it’s something that auditors tend to learn
on the job. Companies’ internal controls tend to evolve over time in response
to real or feared incidents, whether in the company’s own experience or its
auditors’. In the next section, I try to distill into a few principles the experience
gained from several years working at the coalface in banking and consultancy,
and more recently on our university’s finance and other committees.

10.2.2

Designing Internal Controls

Over the years, a number of standards have been put forward by the accountancy profession, by stock markets and by banking regulators, about how
bookkeeping and internal control systems should be designed. In the USA, for
example, there is the Committee of Sponsoring Organizations (COSO), a group of
U.S. accounting and auditing bodies [318]. This self-regulation failed to stop
the excesses of the dotcom era, and following the collapse of Enron there was
intervention from U.S. lawmakers in the form of the Sarbanes-Oxley Act of
2002. It protects whistleblowers (the main source of information on serious
insider fraud), and its section 404 makes managers responsible for maintaining
‘adequate internal control structure and procedures for financial reporting’.
It also demands that auditors attest to the management’s assessment of these
controls and disclose any ‘material weaknesses’. CEOs also have to certify
the truthfulness of financial statements. There was also the Gramm-LeachBliley Act of 1999, which liberalised bank regulation in many respects but

10.2 How Bank Computer Systems Work

which obliged banks to have security mechanisms to protect information from
foreseeable threats in security and integrity. Along with HIPAA in the medical
sector, Gramm-Leach-Bliley and Sarbanes-Oxley have driven much of the
investment in information security and internal control over the early years of
the 21st century. (Other countries have equivalents; in the UK it’s the Turnbull
Guidance from the Financial Reporting Council.) I’ll return to them and look
in more detail at the policy aspects in Part III.
In this section, my concern is with the technical aspects. Modern risk
management systems typically require a company to identify and assess its
risks, and then build controls to mitigate them. A company’s risk register might
contain many pages of items such as ‘insider makes large unauthorised bank
transaction’. Some of these will be mitigated using non-technical measures
such as insurance, but others will end up in your lap. So how do you engineer
away a problem like this?
There are basically two kinds of separation of duty policy: dual control and
functional separation.
In dual control, two or more staff members must act together to authorize
a transaction. The classic military example is in nuclear command systems,
which may require two officers to turn their keys simultaneously in consoles
that are too far apart for any single person to reach both locks. I’ll discuss
nuclear command and control further in a later chapter. The classic civilian
example is when a bank issues a letter of guarantee, which will typically
undertake to carry the losses should a loan made by another bank go sour.
Guarantees are particularly prone to fraud; if you can get bank A to guarantee
a loan to your business from bank B, then bank B is supervising your account
while bank A’s money is at risk. A dishonest businessmen with a forged or
corruptly-obtained guarantee can take his time to plunder the loan account
at bank B, with the alarm only being raised when he absconds and bank B asks
bank A for the money. If a single manager could issue such an instrument, the
temptation would be strong. I’ll discuss this further in section 10.3.2.
With functional separation of duties, two or more different staff members act
on a transaction at different points in its path. The classic example is corporate
purchasing. A manager takes a purchase decision and tells the purchasing
department; a clerk there raises a purchase order; the store clerk records the
goods’ arrival; an invoice arrives at accounts; the accounts clerk correlates it
with the purchase order and the stores receipt and raises a check; and the
accounts manager signs the check.
However, it doesn’t stop there. The manager now gets a debit on her monthly
statement for that internal account, her boss reviews the accounts to make sure
the division’s profit targets are likely to be met, the internal audit department
can descend at any time to audit the division’s books, and when the external
auditors come in once a year they will check the books of a randomly selected

321

322

Chapter 10

■

Banking and Bookkeeping

sample of departments. Finally, when frauds are discovered, the company’s
lawyers may make vigorous efforts to get the money back.
So the model can be described as prevent — detect — recover. The level of
reliance placed on each of these three legs will depend on the application.
Where detection may be delayed for months or years, and recovery may
therefore be very difficult — as with corrupt bank guarantees — it is prudent
to put extra effort into prevention, using techniques such as dual control.
Where prevention is hard, you should see to it that detection is fast enough,
and recovery vigorous enough, to provide a deterrent effect. The classic
example here is that bank tellers can quite easily take cash, so you need to
count the money every day and catch them afterwards.
Bookkeeping and management control are not only one of the earliest
security systems; they also have given rise to much of management science
and civil law. They are entwined with a company’s business processes, and
exist in its cultural context. In Swiss banks, there are two managers’ signatures
on almost everything, while Americans are much more relaxed. In most
countries’ banks, staff get background checks, can be moved randomly from
one task to another, and are forced to take holidays at least once a year.
This would not be acceptable in the typical university — but in academia the
opportunities for fraud are much less.
Designing an internal control system is hard because it’s a highly interdisciplinary problem. The financial controllers, the personnel department, the
lawyers, the auditors and the systems people all come at the problem from different directions, offer partial solutions, fail to understand each other’s control
objectives, and things fall down the hole in the middle. Human factors are very
often neglected, and systems end up being vulnerable to helpful subordinates
or authoritarian managers who can cause dual control to fail. It’s important
not just to match the controls to the culture, but also motivate people to use
them; for example, in the better run banks, management controls are marketed
to staff as a means of protecting them against blackmail and kidnapping.
Security researchers have so far focused on the small part of the problem
which pertains to creating dual control (or in general, where there are more
than two principals, shared control) systems. Even this is not at all easy. For
example, rule 9 in Clark-Wilson says that security officers can change access
rights — so what’s to stop a security officer creating logons for two managers
and using them to send all the bank’s money to Switzerland?
In theory you could use cryptography, and split the signing key between
two or more principals. In a Windows network, the obvious way to manage
things is to put users in separately administered domains. With a traditional
banking system using the mainframe operating system MVS, you can separate
duties between the sysadmin and the auditor; the former can do anything he
wishes, except finding out which of his activities the latter is monitoring [159].
But in real life, dual control is hard to do end-to-end because there are

10.2 How Bank Computer Systems Work

many system interfaces that provide single points of failure, and in any case
split-responsibility systems administration is tedious.
So the practical answer is that most bank sysadmins are in a position to do
just this type of fraud. Some have tried, and where they fall down is when the
back-office balancing controls set off the alarm after a day or two and money
laundering controls stop him getting away with very much. I’ll discuss this
further in section 10.3.2. The point to bear in mind here is that serial controls
along the prevent — detect — recover model are usually more important than
shared control. They depend ultimately on some persistent state in the system
and are in tension with programmers’ desire to keep things simple by making
transactions atomic.
There are also tranquility issues. For example, could an accountant knowing
that he was due to be promoted to manager tomorrow end up doing both
authorizations on a large transfer? A technical fix for this might involve a
Chinese Wall mechanism supporting a primitive ‘X may do Y only if he hasn’t
done Z’ (‘A manager can confirm a payment only if his name doesn’t appear
on it as the creator’). So we would end up with a number of exclusion and
other rules involving individuals, groups and object labels; once the number
of rules becomes large (as it will in a real bank) we would need a systematic
way of examining this rule set and verifying that it doesn’t have any horrible
loopholes.
In the medium term, banking security policy — like medical security
policy — may find its most convenient expression in using role based access
control, although this will typically be implemented in banking middleware
rather than in an underlying platform such as Windows or Linux. Real systems
will need to manage separation-of-duty policies with both parallel elements,
such as dual control, and serial elements such as functional separation along
a transaction’s path. This argues for the access control mechanisms being near
to the application. But then, of course, they are likely to be more complex,
proprietary, and not so well studied as the mechanisms that come with the
operating system.
One really important aspect of internal control in banking — and in systems
generally — is to minimise the number of ‘sysadmins’, that is, of people with
complete access to the whole system and the ability to change it. For decades
now, the standard approach has been to keep development staff quite separate
from live production systems. A traditional bank in the old days would have
two mainframes, one to run the live systems, with the other being a backup
machine that was normally used for development and testing. Programmers
would create new software that would be tested by a test department and
subject to source code review by internal auditors; once approved this would
be handed off to a change management department that would install it in the
live system at the next upgrade. The live system would be run by an operations

323

324

Chapter 10

■

Banking and Bookkeeping

team with no access to compilers, debuggers or other tools that would let them
alter live code or data.
In theory this prevents abuse by programmers, and in practice it can work
fairly well. However there are leaks. First, there are always some sysadmins
who need full access in order to do their jobs; and second, there are always
emergencies. The ATM system goes down at the weekend, and the ATM
team’s duty programmer is given access to the live system from home so
she can fix the bug. You audit such accesses as well as you can, but it’s still
inevitable that your top sysadmins will be so much more knowledgeable than
your auditors that they could do bad things if they really wanted to. Indeed,
at banks I’ve helped with security, you might find that there are thirty or
forty people whom you just have to trust — the CEO, the chief dealer, the top
sysadmins and a number of others. It’s important to know who these people
are, and to minimise their numbers. Pay them well — and watch discreetly to
see if they start spending even more.
A final remark on dual control is that it’s often not adequate for transactions
involving more than one organization, because of the difficulties of dispute
resolution: ‘My two managers say the money was sent!’ ‘But my two say it
wasn’t!’

10.2.3

What Goes Wrong

Theft can take a variety of forms, from the purely opportunist to clever insider
frauds; but the experience is that most thefts from the average company are due
to insiders. There are many surveys; a typical one, by accountants Ernst and
Young, reports that 82% of the worst frauds were committed by employees;
nearly half of the perpetrators had been there over five years and a third of
them were managers [1162].
Typical computer crime cases include:
Paul Stubbs, a password reset clerk at HSBC, conspired with persons
unknown to change the password used by AT&T to access their bank
account with HSBC. The new password was used to transfer £11.8
million — over $20 million — to offshore companies, from which it was
not recovered. Stubbs was a vulnerable young man who had been
employed as a password reset clerk after failing internal exams; the court
took mercy on him and he got away with five years [975]. It was alleged
that an AT&T employee had conspired to cover up the transactions, but
that gentleman was acquitted.
A bank had a system of suspense accounts, which would be used temporarily if one of the parties to a transaction could not be identified (such
as when an account number was entered wrongly on a funds transfer).
This was a workaround added to the dual control system to deal with

10.2 How Bank Computer Systems Work

transactions that got lost or otherwise couldn’t be balanced immediately.
As it was a potential vulnerability, the bank had a rule that suspense
accounts would be investigated if they were not cleared within three
days. One of the clerks exploited this by setting up a scheme whereby
she would post a debit to a suspense account and an equal credit to her
boyfriend’s account; after three days, she would raise another debit to
pay off the first. In almost two years she netted hundreds of thousands
of dollars. (The bank negligently ignored a regulatory requirement that
all staff take at least ten consecutive days’ vacation no more than fifteen
months from the last such vacation.) In the end, she was caught when
she could no longer keep track of the growing mountain of bogus transactions.
A clerk at an education authority wanted to visit relatives in Australia,
and in order to get some money she created a fictitious school, complete with staff whose salaries were paid into her own bank account.
It was only discovered by accident when someone noticed that different
records gave the authority different numbers of schools.
A bank clerk in Hastings, England, noticed that the branch computer
system did not audit address changes. He picked a customer who had
a lot of money in her account and got a statement only once a year; he
then changed her address to his, issued a new ATM card and PIN, and
changed her address back to its original value. He stole £8,600 from her
account, and when she complained she was not believed: the bank maintained that its computer systems were infallible, and so the withdrawals
must have been her fault. The matter was only cleared up when the clerk
got an attack of conscience and started stuffing the cash he’d stolen in
brown envelopes through the branch’s letter box at night. As people
don’t normally give money to banks, the branch manager finally realized
that something was seriously wrong.
Volume crime — such as card fraud — often depends on liability rules.
Where banks can tell customers who complain of fraud to get lost (as in much
of Europe), bank staff know that complaints won’t be investigated properly
or at all, and get careless. Things are better in the USA where Regulation E
places the onus of proof in disputed transaction cases squarely on the bank.
I’ll discuss this in detail in section 10.4 below.
All the really large frauds — the cases over a billion dollars — have involved
lax internal controls. The collapse of Barings Bank is a good example: managers
failed to control rogue trader Nick Leeson, blinded by greed for the bonuses his
apparent trading profits earned them. The same holds true for other financial
sector frauds, such as the Equity Funding scandal, in which an insurance
company’s management created thousands of fake people on their computer
system, insured them, and sold the policies on to reinsurers; and frauds in

325

326

Chapter 10

■

Banking and Bookkeeping

other sectors such as Robert Maxwell’s looting of the Daily Mirror newspaper
pension funds in Britain. (For a collection of computer crime case histories, see
Parker [1005].) Either the victim’s top management were grossly negligent, as
in the case of Barings, or perpetrated the scam, as with Equity Funding and
Maxwell.
The auditors are also a problem. On the one hand, they are appointed
by the company’s managers and are thus extremely bad at detecting frauds
in which the managers are involved; so the assurance that shareholders get
is less than many might have thought. (The legal infighting following the
collapse of Enron destroyed its auditors Arthur Andersen and thus reduced
the ‘big five’ audit firms to the ‘big four’; now auditors go out of their
way to avoid liability for fraud.) Second, there were for many years huge
conflicts of interest, as accountants offered cheap audits in order to get
their foot in the door, whereupon they made their real money from systems
consultancy. (This has been greatly restricted since Enron.) Third, the big
audit firms have their own list of favourite controls, which often bear little
relationship to the client’s real risks, and may even make matters worse.
For example, our university’s auditors nag us every year to get all our staff
to change their passwords every month. This advice is wrong, for reasons
explained in Chapter 2 — so every year we point this out and challenge them
to justify their advice. But they seem incapable of learning, and they have
no incentive to: they can be expected to nitpick, and to ignore any evidence
that a particular nit is unhelpful until long after the evidence has become
overwhelming. While failing to disclose a material weakness could get them
into trouble, at least in the USA, the nitpicking has turned into a bonanza
for them. It’s reckoned that the auditors’ gold-plating of the Sarbanes-Oxley
requirements is costing the average U.S. listed company $2.4m a year in
audit fees, plus 70,000 hours of internal work to ensure compliance; the
total cost of SOX could be as much as $1.4 trillion [412]. (My own advice,
for what it’s worth, is to never use a big-four accountant; smaller firms are
cheaper, and a study done by my student Tyler Moore failed to find any
evidence that companies audited by the Big Four performed better on the
stock market.)
Changing technology also has a habit of eroding controls, which therefore
need constant attention and maintenance. For example, thanks to new systems
for high-speed processing of bank checks, banks in California stopped a
few years ago from honoring requests by depositors that checks have two
signatures. Even when a check has imprinted on it ‘Two Signatures Required’,
banks will honor that check with only one signature [1086]. This might seem
to be a problem for the customer’s security rather than the bank’s, but bank
checks can also be at risk and if something goes wrong even with a merchant
transaction then the bank might still get sued.

10.2 How Bank Computer Systems Work

The lessons to be learned include:
it’s not always obvious which transactions are security sensitive;
maintaining a working security system can be hard in the face of a
changing environment;
if you rely on customer complaints to alert you to fraud, you had better
listen to them;
there will always be people in positions of relative trust who can get
away with a scam for a while;
no security policy will ever be completely rigid. There will always have
to be workarounds for people to cope with real life;
these workarounds naturally create vulnerabilities. So the lower the
transaction error rate, the better.
There will always be residual risks. Managing these residual risks remains
one of the hardest and most neglected of jobs. It means not just technical
measures, such as involving knowledgeable industry experts, auditors and
insurance people in the detailed design, and iterating the design once some
loss history is available. It also means training managers, auditors and others
to detect problems and react to them appropriately. I’ll revisit this in Part III.
The general experience of banks in the English-speaking world is that some
1% of staff are sacked each year. The typical offence is minor embezzlement
with a loss of a few thousand dollars. No-one has found an effective way of
predicting which staff will go bad; previously loyal staff can be thrown off
the rails by shocks such as divorce, or may over time develop a gambling
or alcohol habit. Losing a few hundred tellers a year is simply seen as a
cost of doing business. What banks find very much harder to cope with are
incidents in which senior people go wrong — indeed, in several cases within
my experience, banks have gone to great lengths to avoid admitting that a
senior insider was bent. And risks that managers are unwilling to confront,
they are often unable to control. No-one at Barings even wanted to think that
their star dealer Leeson might be a crook; and pop went the bank.
Finally, it’s not enough, when doing an audit or a security investigation,
to merely check that the books are internally consistent. It’s also important to
check that they correspond to external reality. This was brought home to the
accounting profession in 1938 with the collapse of McKesson and Robbins,
a large, well-known drug and chemical company with reported assets of
$100m1 . It turned out that 20% of the recorded assets and inventory were
nonexistent. The president, Philip Musica, turned out to be an impostor with
1 In 2007 dollars, that’s $1.4bn if you deflate by prices, $3bn if you deflate by unskilled wages and
over $15bn by share of GDP.

327

328

Chapter 10

■

Banking and Bookkeeping

a previous fraud conviction; with his three brothers, he inflated the firm’s
figures using a fake foreign drug business involving a bogus shipping agent
and a fake Montreal bank. The auditors, who had accepted the McKesson
account without making enquiries about the company’s bosses, had failed to
check inventories, verify accounts receivable with customers, or think about
separation of duties within the company [1082]. The lessons from that incident
clearly weren’t learned well enough, as the same general things continue to
happen regularly and on all sorts of scales from small firms and small branches
of big ones, to the likes of Enron.
So if you ever have responsibility for security in a financial (or other) firm,
you should think hard about which of your managers could defraud your
company by colluding with customers or suppliers. Could a branch manager
be lending money to a dodgy business run by his cousin against forged
collateral? Could he have sold life-insurance policies to nonexistent people
and forged their death certificates? Could an operations manager be taking
bribes from a supplier? Could one of your call-center staff be selling customer
passwords to the Mafia? Lots of things can and do go wrong; you have to figure
out which of them matter, and how you get to find out. Remember: a trusted
person is one who can damage you. Who can damage you, and how? This is the
basic question that a designer of internal controls must be constantly asking.

10.3

Wholesale Payment Systems

When people think of electronic bank fraud, they often envisage a Hollywood
scene in which crafty Russian hackers break a bank’s codes and send multimillion dollar wire transfers to tax havens. Systems for transferring money
electronically are indeed an occasional target of sophisticated crime, and have
been for a century and a half, as I noted earlier in section 5.2.4 when I discussed
test key systems.
By the early 1970s, bankers started to realise that this worthy old Victorian
system was due for an overhaul.
First, most test-key systems were vulnerable in theory at least to cryptanalysis; someone who observed a number of transactions could gradually work
out the key material.
Second, although the test key tables were kept in the safe, there was nothing
really to stop staff members working out tests for unauthorised messages at
the same time as a test for an authorised message. In theory, you might require
that two staff members retrieve the tables from the safe, sit down at a table
facing each other and perform the calculation. However, in practice people
would work sequentially in a corner (the tables were secret, after all) and even
if you could compel them to work together, a bent employee might mentally
compute the test on an unauthorized message while overtly computing the

10.3 Wholesale Payment Systems

test on an authorized one. So, in reality, test key schemes didn’t support dual
control. Having tests computed by one staff member and checked by another
doubled the risk rather than reducing it. (There are ways to do dual control
with manual authenticators, such as by getting the two staff members to work
out different codes using different tables; and there are other techniques, used
in the control of nuclear weapons, which I’ll discuss in 13.4.)
Third, there was a big concern with cost and efficiency. There seemed
little point in having the bank’s computer print out a transaction in the telex
room, having a test computed manually, then composing a telex to the other
bank, then checking the test, and then entering it into the other bank’s
computer. Errors were much more of a problem than frauds, as the telex
operators introduced typing errors. Customers who got large payments into
their accounts in error sometimes just spent the money, and in one case an
erroneous recipient spent some of his windfall on clever lawyers, who helped
him keep it. This shocked the industry. Surely the payments could flow directly
from one bank’s computer to another?

10.3.1 SWIFT
The Society for Worldwide International Financial Telecommunications
(SWIFT) was set up in the 1970s by a consortium of banks to provide a
more secure and efficient means than telex of sending payment instructions
between member banks. It can be thought of as an email system with built-in
encryption, authentication and non-repudiation services. It’s important not
just because it’s used to ship trillions of dollars round the world daily, but
because its design has been copied in systems processing many other kinds of
intangible asset, from equities to bills of lading.
The design constraints are interesting. The banks did not wish to trust SWIFT,
in the sense of enabling dishonest employees there to forge transactions. The
authenticity mechanisms had to be independent of the confidentiality mechanisms, since at the time a number of countries (such as France) forbade the
civilian use of cryptography for confidentiality. The non-repudiation functions had to be provided without the use of digital signatures, as these
hadn’t been invented yet. Finally, the banks had to be able to enforce
Clark-Wilson type controls over interbank transactions. (Clark-Wilson also
hadn’t been invented yet but its components, dual control, balancing, audit
and so on, were well enough established.)
The SWIFT design is summarized in Figure 10.2. Authenticity of messages
was assured by computing a message authentication code (MAC) at the
sending bank and checking it at the receiving bank. The keys for this MAC
used to be managed end-to-end: whenever a bank set up a relationship
overseas, the senior manager who negotiated it would exchange keys with
his opposite number, whether in a face-to-face meeting or afterwards by post
to each others’ home addresses. There would typically be two key components

329

330

Chapter 10

■

Banking and Bookkeeping

to minimize the risk of compromise, with one sent in each direction (since
even if a bank manager’s mail is read in his mailbox by a criminal at one end,
it’s not likely to happen at both). The key was not enabled until both banks
confirmed that it had been safely received and installed.
This way, SWIFT had no part in the message authentication; so long as the
authentication algorithm in use was sound, none of their staff could forge a
transaction. The authentication algorithm used is supposed to be a trade secret;
but as banks like their security mechanisms to be international standards, a
natural place to look might be the algorithm described in ISO 8731 [1094]. In
this way, they got the worst of all possible worlds: the algorithm was fielded
without the benefit of public analysis but got it later once it was expensive
to change! An attack was found on the ISO 8731 message authentication
algorithm and published in [1040] but, fortunately for the industry, it takes
over 100,000 messages to recover a key — which is too large for a practical
attack on a typical system that is used prudently.
Although SWIFT itself was largely outside the trust perimeter for message
authentication, it did provide a non-repudiation service. Banks in each country
sent their messages to a Regional General Processor (RGP) which logged them
and forwarded them to SWIFT, which also logged them and sent them on
to the recipient via the RGP in his country, which also logged them. The
RGPs were generally run by different facilities management firms. Thus a
bank (or a crooked bank employee) wishing to dishonestly repudiate a done
transaction would have to subvert not just the local SWIFT application and
its surrounding controls, but also two independent contractors in different
countries. Note that the repudiation control from multiple logging is better
than the integrity control. A bent bank wanting to claim that a transaction had
been done when it hadn’t could always try to insert the message between the
other bank and their RGP; while a bent bank employee would probably just
insert a bogus incoming message directly into a local system. So logs can be a
powerful way of making repudiation difficult, and are much easier for judges
to understand than cryptography.

Logs

RGP

Key

Swift

RGP

Bank

Bank

Branch

Branch

Figure 10.2: Architecture of SWIFT

Key

10.3 Wholesale Payment Systems

Confidentiality depended on line encryption devices between the banks
and the RGP node, and between these nodes and the main SWIFT processing
sites. Key management was straightforward at first. Keys were hand carried
in EEPROM cartridges between the devices at either end of a leased line. In
countries where confidentiality was illegal, these devices could be omitted
without impairing the authenticity and non-repudiation mechanisms.
Dual control was provided either by the use of specialized terminals (in
small banks) or by mainframe software packages which could be integrated
with a bank’s main production system. The usual method of operation is
to have three separate staff to do a SWIFT transaction: one to enter it, one to
check it, and one to authorize it. (As the checker can modify any aspect of
the message, this really only gives dual control, not triple control — and the
programmers who maintain the interface can always attack the system there).
Reconciliation was provided by checking transactions against daily statements
received electronically from correspondent banks. This meant that someone
who managed to get a bogus message into the system would sound an alarm
within two or three days.

10.3.2 What Goes Wrong
SWIFT I ran for twenty years without a single report of external fraud. In the
mid 1990s, it was enhanced by adding public key mechanisms; MAC keys are
now shared between correspondent banks using public key cryptography and
the MACs themselves may be further protected by a digital signature. The key
management mechanisms have been ensconced as ISO standard 11166, which
in turn has been used in other systems (such as CREST, which is used by banks
and stockbrokers to register and transfer UK stocks and shares). There has
been some debate over the security of this architecture [73, 1094]. Quite apart
from the centralization of trust brought about by the adoption of public key
cryptography — in that the central certification authority can falsely certify
a key as belonging to a bank when it doesn’t — CREST adopted 512-bit public
keys, and these are too short: as I mentioned in the chapter on cryptology, at
least one RSA public key of this length has been factored surreptitiously by a
group of students [24].
However the main practical attacks on such systems have not involved
the payment mechanisms themselves. The typical attack comes from a bank
programmer inserting a bogus message into the processing queue. It usually
fails because he does not understand the other controls in the system, or the
procedural controls surrounding large transfers. For example, banks maintain
accounts with each other, so when bank A sends money to a customer of
bank B it actually sends an instruction ‘please pay this customer the following
sum out of our account with you’. As these accounts have both balances and
credit limits, large payments aren’t processed entirely automatically but need
intervention from the dealing room to ensure that the needed currency or

331

332

Chapter 10

■

Banking and Bookkeeping

credit line is made available. So transfers over a million dollars or so tend to
need managerial interventions of which technical staff are ignorant; and there
are also filters that look for large transactions so that the bank can report them
to the money-laundering authorities if need be. There is also the common-sense
factor, in that anyone who opens a bank account, receives a large incoming
wire transfer and then starts frantically moving money out again used to need
a very convincing excuse. (Common sense has become less of a backstop since
9/11 as customer due diligence and anti-money-laundering rules have become
both formalised and onerous; bank staff rely more on box-ticking, which has
made life easier for the bad guys [55].) In any case, the programmer who
inserts a bogus transaction into the system usually gets arrested when he turns
up to collect the cash. If your life’s goal is a career in bank fraud, you’re better
off getting an accounting or law degree and working in a loans office rather
than messing about with computers.
Other possible technical attacks, such as inserting Trojan software into the
PCs used by bank managers to initiate transactions, wiretapping the link from
the branch to the bank mainframe, subverting the authentication protocol used
by bank managers to log on, and even inserting a bogus transaction in the
branch LAN causing it to appear on the relevant printer — would also run
up against the business-process controls. In fact, most large scale bank frauds
which ‘worked’ have not used technical attacks but exploited procedural
vulnerabilities.
The classic example is a letter of guarantee. It is common enough for a
company in one country to guarantee a loan to a company in another.
This can be set up as a SWIFT message, or even a paper letter. But as
no cash changes hands at the time, the balancing controls are inoperative. If a forged guarantee is accepted as genuine, the ‘beneficiary’ can
take his time borrowing money from the accepting bank, laundering it,
and disappearing. Only when the victim bank realises that the loan has
gone sour and tries to call in the guarantee is the forgery discovered.
An interesting fraud of a slightly different type took place in 1986
between London and Johannesburg. At that time, the South African
government operated two exchange rates, and in one bank the manager responsible for deciding which rate applied to each transaction
conspired with a rich man in London. They sent money out to Johannesburg at an exchange rate of seven Rand to the Pound, and back
again the following day at four. After two weeks of this, the authorities
became suspicious, and the police came round. On seeing them in the
dealing room, the manager fled without stopping to collect his jacket,
drove over the border to Swaziland, and flew via Nairobi to London.
There, he boasted to the press about how he had defrauded the wicked
apartheid system. As the UK has no exchange control, exchange control

10.4 Automatic Teller Machines

fraud isn’t an offence and so he couldn’t be extradited. The conspirators got away with millions, and the bank couldn’t even sue them.
Perhaps the best known money transfer fraud occurred in 1979 when
Stanley Rifkin, a computer consultant, embezzled over ten million
dollars from Security Pacific National Bank. He got round the money
laundering controls by agreeing to buy a large shipment of diamonds
from a Russian government agency in Switzerland. He got the transfer into the system by observing an authorization code used internally
when dictating transfers to the wire transfer department, and simply
used it over the telephone (a classic example of dual control breakdown
at a system interface). He even gave himself extra time to escape by
doing the deal just before a U.S. bank holiday. Where he went wrong
was in not planning what to do after he collected the stones. If he’d
hid them in Europe, gone back to the USA and helped investigate the
fraud, he might well have got away with it; as it was, he ended up on
the run and got caught.
The system design lesson is unchanged: one must always be alert to things
which defeat the dual control philosophy. However, as time goes on we have
to see it in a broader context. Even if we can solve the technical problems of
systems administration, interfaces and so on, there’s still the business process
problem of what we control — quite often critical transactions don’t appear as
such at a casual inspection.

10.4

Automatic Teller Machines

Another set of lessons about the difficulties and limitations of dual control emerges from studying the security of automatic teller machines (ATMs).
ATMs, also known as cash machines, have been one of the most influential
technological innovations of the 20th century.
ATMs were the first large-scale retail transaction processing systems. They
were devised in 1938 by the inventor Luther Simjian, who also thought up
the teleprompter and the self-focussing camera. He persuaded Citicorp to
install his ‘Bankamat’ machine in New York in 1939; they withdrew it after
six months, saying ‘the only people using the machines were a small number
of prostitutes and gamblers who didn’t want to deal with tellers face to
face’ [1168]. Its commercial introduction dates to 1967, when a machine made
by De La Rue was installed by Barclays Bank in Enfield, London. The world
installed base is now thought to be about 1,500,000 machines. The technology
developed for them is now also used in card payment terminals in shops.
Modern block ciphers were first used on a large scale in ATM networks:
they are used to generate and verify PINs in secure hardware devices located
within the ATMs and at bank computer centres. This technology, including

333

334

Chapter 10

■

Banking and Bookkeeping

block ciphers, tamper-resistant hardware and the supporting protocols, ended
up being used in many other applications from postal franking machines to
lottery ticket terminals. In short, ATMs were the ‘killer app’ that got modern
commercial cryptology and retail payment technology off the ground.

10.4.1

ATM Basics

Most ATMs operate using some variant of a system developed by IBM for its
3624 series cash machines in the late 1970s. The card contains the customer’s
primary account number, PAN. A secret key, called the ‘PIN key’, is used to
encrypt the account number, then decriminalize it and truncate it. The result
of this operation is called the ‘natural PIN’; an offset can be added to it in
order to give the PIN which the customer must enter. The offset has no real
cryptographic function; it just enables customers to choose their own PIN. An
example of the process is shown in Figure 10.3.
In the first ATMs to use PINs, each ATM contained a copy of the PIN key
and each card contained the offset as well as the primary account number.
Each ATM could thus verify all customer PINs. Early ATMs also operated
offline; if your cash withdrawal limit was $500 per week, a counter was kept
on the card. In recent years networks have become more dependable and
ATMs have tended to operate online only, which simplifies the design; the
cash counters and offsets have vanished from magnetic strips and are now
kept on servers. In the last few years, magnetic strips have been supplemented
with smartcard chips in some countries, especially in Europe; I will describe
the smartcard systems later. However the basic principle remains: PINs are
generated and protected using cryptography.
Dual control is implemented in this system using tamper-resistant hardware.
A cryptographic processor, also known as a security module, is kept in the bank’s
server room and will perform a number of defined operations on customer
PINs and on related keys in ways that enforce a dual-control policy. This
includes the following.
1. Operations on the clear values of customer PINs, and on the keys needed
to compute them or used to protect them, are all done in tamper-resistant
Account number PAN:
PIN key KP:
Result of DES {PAN}KP :
{N}KP decimalized:
Natural PIN:
Offset:
Customer PIN:

8807012345691715
FEFEFEFEFEFEFEFE
A2CE126C69AEC82D
0224126269042823
0224
6565
6789

Figure 10.3: IBM method for generating bank card PINs

10.4 Automatic Teller Machines

hardware and the clear values are never made available to any single
member of the bank’s staff.
2. Thus, for example, the cards and PINs are sent to the customer via separate channels. The cards are personalized in a facility with embossing
and mag strip printing machinery, and the PIN mailers are printed in a
separate facility containing a printer attached to a security module.
3. A terminal master key is supplied to each ATM in the form of two printed
components, which are carried to the branch by two separate officials,
input at the ATM keyboard, and combined to form the key. Similar procedures (but with three officials) are used to set up master keys between
banks and network switches such as VISA.
4. If ATMs are to perform PIN verification locally, then the PIN key is
encrypted under the terminal master key and then sent to the ATM.
5. If the PIN verification is to be done centrally over the network, the PIN
is encrypted under a key set up using the terminal master key. It will
then be sent from the ATM to a central security module for checking.
6. If the bank’s ATMs are to accept other banks’ cards, then its security
modules use transactions that take a PIN encrypted under an ATM key,
decrypt it and re-encrypt it for its destination, such as using a key shared
with VISA. This PIN translation function is done entirely within the hardware security module, so that clear values of PINs are never available to
the bank’s programmers. VISA will similarly decrypt the PIN and reencrypt it using a key shared with the bank that issued the card, so that
the PIN can be verified by the security module that knows the relevant
PIN key.
During the 1980s and 1990s, hardware security modules became more and
more complex, as ever more functionality got added to support more complex
financial applications from online transactions to smartcards. An example of
a leading product is the IBM 4758 — this also has the virtue of having its
documentation available publicly online for study (see [641] for the command
set and [1195] for the architecture and hardware design). We’ll discuss this
later in the chapter on tamper resistance.
But extending the dual control security policy from a single bank to tens
of thousands of banks worldwide, as modern ATM networks do, was not
completely straightforward.
When people started building ATM networks in the mid 1980s, many
banks used software encryption rather than hardware security modules to support ATMs. So in theory, any bank’s programmers might
get access to the PINs of any other bank’s customers. The remedy was
to push through standards for security module use. In many countries

335

336

Chapter 10

■

Banking and Bookkeeping

(such as the USA), these standards were largely ignored; but even where
they were respected, some banks continued using software for transactions involving their own customers. So some keys (such as those used
to communicate with ATMs) had to be available in software too, and
knowledge of these keys could be used to compromise the PINs of other
banks’ customers. (I’ll explain this in more detail later.) So the protection
given by the hardware TCB was rarely complete.
It is not feasible for 10,000 banks to share keys in pairs, so each bank
connects to a switch provided by an organization such as VISA or Cirrus, and the security modules in these switches translate the traffic. The
switches also do accounting, and enable banks to settle their accounts for
each day’s transactions with all the other banks in the system by means
of a single electronic debit or credit. The switch is highly trusted, and
if something goes wrong there the consequences can be severe. In one
case, there turned out to be not just security problems but also dishonest
staff. The switch manager ended up a fugitive from justice, and the bill
for remediation was in the millions. In another case, a Y2K-related software upgrade at a switch was bungled, with the result that cardholders
in one country found that for a day or two they could withdraw money
even if their accounts were empty. This also led to a very large bill.
Corners are cut to reduce the cost of dealing with huge transaction
volumes. For example, it is common for authentication of authorization responses to be turned off. So anyone with access to the network
can cause a given ATM to accept any card presented to it, by simply
replaying a positive authorization response. Network managers claim
that should a fraud ever start, then the authentication can always be
turned back on. This might seem reasonable; attacks involving manipulated authorization responses are very rare. Similarly, after UK banks
put smartcard chips into bank cards, some of them kept on accepting magnetic-strip transactions, so that a card with a broken chip
would still work so long as the magnetic strip could be read. But such
shortcuts — even when apparently reasonable on grounds of risk and
cost — mean that a bank which stonewalls customer complaints by
saying its ATM network is secure, and so the transaction must be the
customer’s fault, is not telling the truth. This may lay the bank and its
directors open to fraud charges. What’s more, changing the network’s
modus operandi suddenly in response to a fraud can be difficult; it
can unmask serious dependability problems or lead to unacceptable
congestion. This brings home the late Roger Needham’s saying that
‘optimization is the process of taking something which works, and
replacing it by something which doesn’t quite but is cheaper’.

10.4 Automatic Teller Machines

There are many other ways in which ATM networks can be attacked in
theory, and I’ll discuss a number of them later in the context of interface
security: the design of the hardware security modules that were in use for
decades was so poor that programmers could extract PINs and keys simply
by issuing suitable sequences of commands to the device’s interface with
the server, without having to break either the cryptographic algorithms or the
hardware tamper-resistance. However, one of the interesting things about
these systems is that they have now been around long enough, and have been
attacked enough by both insiders and outsiders, to give us a lot of data points
on how such systems fail in practice.

10.4.2 What Goes Wrong
ATM fraud is an interesting study as this is a mature system with huge
volumes and a wide diversity of operators. There have been successive waves
of ATM fraud, which have been significant since the early 1990s. In each wave,
a set of vulnerabilities was exploited and then eventually fixed; but the rapidly
growing scale of payment card operations opened up new vulnerabilities.
There is a fascinating interplay between the technical and regulatory aspects
of protection.
The first large wave of fraud lasted from perhaps 1990–96 and exploited
the poor implementation and management of early systems. In the UK, one
prolific fraudster, Andrew Stone, was convicted three times of ATM fraud,
the last time getting five and a half years in prison. He got involved in fraud
when he discovered by chance the ‘encryption replacement’ trick I discussed
in the chapter on protocols: he changed the account number on his bank
card to his wife’s and found by chance that he could take money out of her
account using his PIN. In fact, he could take money out of any account at that
bank using his PIN. This happened because his bank (and at least two others)
wrote the encrypted PIN to the card’s magnetic strip without linking it to the
account number in any robust way (for example, by using the ‘offset’ method
described above). His second method was ‘shoulder surfing’: he’d stand in
line behind a victim, observe the entered PIN, and pick up the discarded ATM
slip. Most banks at the time printed the full account number on the slip, and a
card would work with no other correct information on it.
Stone’s methods spread via people he trained as his accomplices, and via a
‘Howto’ manual he wrote in prison. Some two thousand victims of his (and
other) frauds banded together to bring a class action against thirteen banks to
get their money back; the banks beat this on the technical legal argument that
the facts in each case were different. I was an expert in this case, and used it to
write a survey of what went wrong [33] (there is further material in [34]). The
fraud spread to the Netherlands, Italy and eventually worldwide, as criminals

337

338

Chapter 10

■

Banking and Bookkeeping

learned a number of simple hacks. Here I’ll summarize the more important
and interesting lessons we learned.
The engineers who designed ATM security systems in the 1970s and 1980s
(of whom I was one) had assumed that criminals would be relatively sophisticated, fairly well-informed about the system design, and rational in their choice
of attack methods. In addition to worrying about the many banks which were
slow to buy security modules and implementation loopholes such as omitting
authentication codes on authorization responses, we agonized over whether
the encryption algorithms were strong enough, whether the tamper-resistant
boxes were tamper-resistant enough, and whether the random number generators used to manufacture keys were random enough. We knew we just
couldn’t enforce dual control properly: bank managers considered it beneath
their dignity to touch a keyboard, so rather than entering the ATM master
key components themselves after a maintenance visit, most of them would
just give both key components to the ATM engineer. We wondered whether a
repairman would get his hands on a bank’s PIN key, forge cards in industrial
quantities, close down the whole system, and wreck public confidence in
electronic banking.
The great bulk of the actual ‘phantom withdrawals’, however, appeared to
have one of the following three causes:
Simple processing errors account for a lot of disputes. With U.S. customers making something like 5 billion ATM withdrawals a year, even
a system that only makes one error per hundred thousand transactions
will give rise to 50,000 disputes a year. In practice the error rate seems to
lie somewhere between 1 in 10,000 and 1 in 100,000. One source of errors
we tracked down was that a large bank’s ATMs would send a transaction again if the network went down before a confirmation message was
received from the mainframe; periodically, the mainframe itself crashed
and forgot about open transactions. We also found customers whose
accounts were debited with other customers’ transactions, and other
customers who were never debited at all for their card transactions. (We
used to call these cards ‘directors’ cards’ and joked that they were issued
to bank directors.)
Thefts from the mail were also huge. They are reckoned to account for
30% of all UK payment card losses, but most banks’ postal control
procedures have always been dismal. For example, when I moved to
Cambridge in February 1992 I asked my bank for an increased card limit:
the bank sent not one, but two, cards and PINs through the post. These
cards arrived only a few days after intruders had got hold of our apartment block’s mail and torn it up looking for valuables. It turned out that
this bank did not have the systems to deliver a card by registered post.
(I’d asked them to send the card to the branch for me to collect but the

10.4 Automatic Teller Machines

branch staff had simply re-addressed the envelope to me.) Many banks
now make you phone a call center to activate a card before you can use
it. This made a dent in the fraud rates.
Frauds by bank staff appeared to be the third big cause of phantoms.
We mentioned the Hastings case in section 10.2.3 above; there are many
others. For example, in Paisley, Scotland, an ATM repairman installed
a portable computer inside an ATM to record customer card and PIN
data and then went on a spending spree with forged cards. In London,
England, a bank stupidly used the same cryptographic keys in its live
and test systems; maintenance staff found out that they could work out
customer PINs using their test equipment, and started offering this as a
service to local criminals at £50 a card. Insider frauds were particularly
common in countries like Britain where the law generally made the customer pay for fraud, and rarer in countries like the USA where the bank
paid; British bank staff knew that customer complaints wouldn’t be
investigated carefully, so they got lazy, careless, and sometimes bent.
These failures are all very much simpler and more straightforward than the
ones we’d worried about. In fact, the only fraud we had worried about and that
actually happened to any great extent was on offline processing. In the 1980s,
many ATMs would process transactions while the network was down, so as
to give 24-hour service; criminals — especially in Italy and later in England
too — learned to open bank accounts, duplicate the cards and then ‘jackpot’ a
lot of ATMs overnight when the network was down [775]. This forced most
ATM operations to be online-only by 1994.
However, there were plenty of frauds that happened in quite unexpected
ways. I already mentioned the Utrecht case in section 3.5, where a tap on a
garage point-of-sale terminal was used to harvest card and PIN data; and
Stone’s ‘encryption replacement’ trick. There were plenty more.
Stone’s shoulder-surfing trick of standing in an ATM queue, observing
a customer’s PIN, picking up the discarded ticket and copying the data
to a blank card, was not in fact invented by him. It was first reported
in New York in the mid 1980s; and it was still working in the Bay Area
in the mid 1990s. By then it had been automated; Stone (and Bay area
criminals) used video cameras with motion sensors to snoop on PINs,
whether by renting an apartment overlooking an ATM or even parking a
rented van there. Visual copying is easy to stop: the standard technique
nowadays is to print only the last four digits of the account number on
the ticket, and there’s also a three-digit ‘card verification value’ (CVV)
on the magnetic strip that should never be printed. Thus even if the villain’s camera is good enough to read the account number and expiry
date from the front of the card, a working copy can’t be made. (The CVV

339

340

Chapter 10

■

Banking and Bookkeeping

is like the three-digit security code on the signature strip, but different
digits; each is computed from the account number and expiry date by
encrypting them with a suitable key). Surprisingly, it still happens; I
have a letter from a UK bank dated May 2007 claiming that a series of
terrorist-related ATM frauds were perpetrated using closed-circuit TV.
This amounts to an admission that the CVV is not always checked.
There were some losses due to programming errors by banks. One small
institution issued the same PIN to all its customers; another bank’s cash
machines had the feature that when a telephone card was entered at an
ATM, it believed that the previous card had been inserted again. Crooks
stood in line, observed customers’ PINs, and helped themselves.
There were losses due to design errors by ATM vendors. One
model, common in the 1980s, would output ten banknotes from
the lowest denomination non-empty cash drawer, whenever a certain fourteen digit sequence was entered at the keyboard. One
bank printed this sequence in its branch manual, and three years
later there was a sudden spate of losses. All the banks using the
machine had to rush out a patch to disable the test transaction.
And despite the fact that I documented this in 1993, and again
in the first edition of this book in 2001, similar incidents are still
reported in 2007. Some makes of ATM used in stores can be reprogrammed into thinking that they are dispensing $1 bills when
in fact they’re dispensing twenties; it just takes a default master password that is printed in widely-available online manuals. Any passer-by who knows this can stroll up to the machine,
reset the bill value, withdraw $400, and have his account debited
with $20. The store owners who lease the machines are not told of
the vulnerability, and are left to pick up the tab [1037].
Several banks thought up check-digit schemes to enable PINs to be
checked by offline ATMs without having to give them the bank’s
PIN key. For example, customers of one British bank get a credit
card PIN with digit one plus digit four equal to digit two plus digit
three, and a debit card PIN with one plus three equals two plus four.
Crooks found they could use stolen cards in offline devices by entering a PIN such as 4455.
Many banks’ operational security procedures were simply dire. In
August 1993, my wife went into a branch of our bank with a witness and
told them she’d forgotten her PIN. The teller helpfully printed her a new
PIN mailer from a printer attached to a PC behind the counter — just
like that! It was not the branch where our account is kept. Nobody knew
her and all the identification she offered was our bank card and her
checkbook. When anyone who’s snatched a handbag can walk in off

10.4 Automatic Teller Machines

the street and get a PIN for the card in it at any branch, no amount of
encryption technology will do much good. (The bank in question has
since fallen victim to a takeover.)
A rapidly growing modus operandi in the early 1990s was to use false
terminals to collect card and PIN data. The first report was from the
USA in 1988; there, crooks built a vending machine which would accept
any card and PIN, and dispense a packet of cigarettes. In 1993, two villains installed a bogus ATM in the Buckland Hills Mall in Connecticut [667, 962]. They had managed to get a proper ATM and a software
development kit for it — all bought on credit. Unfortunately for them,
they decided to use the forged cards in New York, where cash machines
have hidden video cameras, and as they’d crossed a state line they ended
up getting long stretches in Club Fed.
So the first thing we did wrong when designing ATM security systems in the
early to mid 1980s was to worry about criminals being clever, when we should
rather have worried about our customers — the banks’ system designers,
implementers and testers — being stupid. Crypto is usually only part of a
very much larger system. It gets a lot of attention because it’s mathematically
interesting; but as correspondingly little attention is paid to the ‘boring’ bits
such as training, usability, standards and audit, it’s rare that the bad guys have
to break the crypto to compromise a system. It’s also worth bearing in mind
that there are so many users for large systems such as ATM networks that we
must expect the chance discovery and exploitation of accidental vulnerabilities
which were simply too obscure to be caught in testing.
The second thing we did wrong was to not figure out what attacks could
be industrialised, and focus on those. In the case of ATMs, the false-terminal
attack is the one that made the big time. The first hint of organised crime
involvement was in 1999 in Canada, where dozens of alleged Eastern European
organized-crime figures were arrested in the Toronto area for deploying
doctored point-of-sale terminals [85, 152]. The technology has since become
much more sophisticated; ‘skimmers’ made in Eastern Europe are attached to
the throats of cash machines to read the magnetic strip and also capture the
PIN using a tiny camera. I’ll discuss these in more detail in the next section.
Despite attempts to deal with false-terminal attacks by moving from magnetic
strip cards to smartcards, they have become pervasive. They will be difficult
and expensive to eliminate.

10.4.3 Incentives and Injustices
In the USA, the banks have to carry the risks associated with new technology.
This was decided in a historic precedent, Judd versus Citibank, in which bank
customer Dorothy Judd claimed that she had not made some disputed withdrawals and Citibank said that as its systems were secure, she must have done.

341

342

Chapter 10

■

Banking and Bookkeeping

The judge ruled that Citibank’s claim to infallibility was wrong in law, as it
put an unmeetable burden of proof on her, and gave her her money back [674].
The U.S. Federal Reserve incorporated this into ‘Regulation E’, which requires
banks to refund all disputed transactions unless they can prove fraud by
the customer [440]. This has led to some minor abuse — misrepresentations
by customers are estimated to cost the average U.S. bank about $15,000 a
year — but this is an acceptable cost (especially as losses from vandalism are
typically three times as much) [1362].
In other countries — such as the UK, Germany, the Netherlands and Norway — the banks got away for many years with claiming that their ATM
systems were infallible. Phantom withdrawals, they maintained, could not
possibly exist and a customer who complained of one must be mistaken or
lying. This position was demolished in the UK when Stone and a number of
others started being jailed for ATM fraud, as the problem couldn’t be denied
any more. Until that happened, however, there were some rather unpleasant
incidents which got banks a lot of bad publicity [34]. The worst was maybe the
Munden case.
John Munden was one of our local police constables, based in Bottisham,
Cambridgeshire; his beat included the village of Lode where I lived at the
time. He came home from holiday in September 1992 to find his bank account
empty. He asked for a statement, found six withdrawals for a total of £460
which he did not recall making, and complained. His bank responded by
having him prosecuted for attempting to obtain money by deception. It
came out during the trial that the bank’s system had been implemented and
managed in a ramshackle way; the disputed transactions had not been properly
investigated; and all sorts of wild claims were made by the bank, such as that
their ATM system couldn’t suffer from bugs as its software was written in
assembler. Nonetheless, it was his word against the bank’s. He was convicted
in February 1994 and sacked from the police force.
This miscarriage of justice was overturned on appeal, and in an interesting
way. Just before the appeal was due to be heard, the prosecution served up a
fat report from the bank’s auditors claiming that the system was secure. The
defense demanded equal access to the bank’s systems for its own expert.
The bank refused and the court therefore disallowed all its computer evidence — including even its bank statements. The appeal succeeded, and John
got reinstated. But this was only in July 1996 — he’d spent the best part of four
years in limbo and his family had suffered terrible stress. Had the incident happened in California, he could have won enormous punitive damages — a point
bankers should ponder as their systems become global and their customers
can be anywhere.2
2
Recently the same drama played itself out again when Jane Badger, of Burton-on-Trent, England,
was prosecuted for complaining about phantom withdrawals. The case against her collapsed in
January 2008. The bank, which is called Egg, is a subsidiary of Citicorp.

10.5 Credit Cards

The lesson to be drawn from such cases is that dual control is not enough. If
a system is to provide evidence, then it must be able to withstand examination
by hostile experts. In effect, the bank had used the wrong security policy.
What they really needed wasn’t dual control but non-repudiation: the ability
for the principals in a transaction to prove afterwards what happened. This
could have been provided by installing ATM cameras; although these were
available (and are used in some U.S. states), they were not used in Britain.
Indeed, during the 1992–4 wave of ATM frauds, the few banks who had
installed ATM cameras were pressured by the other banks into withdrawing
them; camera evidence could have undermined the stance that the banks took
in the class action that their systems were infallible.
One curious thing that emerged from this whole episode was that although
U.S. banks faced a much fiercer liability regime, they actually spent less on security than UK banks did, and UK banks suffered more fraud.This appears to have
been a moral-hazard effect, and was one of the anomalies that sparked interest
in security economics. Secure systems need properly aligned incentives.

10.5

Credit Cards

The second theme in consumer payment systems is the credit card. For many
years after their invention in the 1950s, credit cards were treated by most banks
as a loss leader with which to attract high-value customers. Eventually, in most
countries, the number of merchants and cardholders reached critical mass and
the transaction volume suddenly took off. In Britain, it took almost twenty
years before most banks found the business profitable; then all of a sudden it
was extremely profitable. Payment systems have strong network externalities,
just like communications technologies or computer platforms: they are twosided markets in which the service provider must recruit enough merchants to
appeal to cardholders, and vice versa. Because of this, and the huge investment
involved in rolling out a new payment system to tens of thousands of banks,
millions of merchants and billions of customers worldwide, any new payment
mechanism is likely to take some time to get established. (The potentially
interesting exceptions are where payment is bundled with some other service,
such as with Google Checkout.)
Anyway, when you use a credit card to pay for a purchase in a store, the
transaction flows from the merchant to his bank (the acquiring bank) which
pays him after deducting a merchant discount of typically 4–5%. If the card was
issued by a different bank, the transaction now flows to a switching center run
by the brand (such as VISA) which takes a commission and passes it to the
issuing bank for payment. Daily payments between the banks and the brands
settle the net cash flows. The issuer also gets a slice of the merchant discount,
but makes most of its money from extending credit to cardholders at rates that
are usually much higher than the interbank rate.

343

344

Chapter 10

10.5.1

■

Banking and Bookkeeping

Fraud

The risk of fraud using stolen cards was traditionally managed by a system
of hot card lists and merchant floor limits. Each merchant gets a local hot card
list — formerly on paper, now stored in his terminal — plus a limit set by
their acquiring bank above which they have to call for authorization. The call
center, or online service, which he uses for this has access to a national hot
card list; above a higher limit, they will contact VISA or MasterCard which
has a complete list of all hot cards being used internationally; and above a still
higher limit, the transaction will be checked all the way back to the card issuer.
Recently, the falling cost of communications has led to many transactions being
authorised all the way back to the issuer, but there are still extensive fallback
processing capabilities. This is because maintaining 99.9999% availability on
a network, plus the capacity to handle peak transaction volumes on the
Wednesday before Thanksgiving and the Saturday just before Christmas, still
costs a whole lot more than the fraud from occasional offline and stand-in
processing.
The introduction of mail order and telephone order (MOTO) transactions in the
1970s meant that the merchant did not have the customer present, and was
not able to inspect the card. What was to stop someone ordering goods using
a credit card number he’d picked up from a discarded receipt?
Banks managed the risk by using the expiry date as a password, lowering
the floor limits, increasing the merchant discount and insisting on delivery to
a cardholder address, which is supposed to be checked during authorization.
But the main change was to shift liability so that the merchant bore the full
risk of disputes. If you challenge an online credit card transaction (or in fact
any transaction made under MOTO rules) then the full amount is immediately
debited back to the merchant, together with a significant handling fee. The
same procedure applies whether the debit is a fraud, a dispute or a return.
A recent development has been the ‘Verified by VISA’ program under which
merchants can refer online credit-card transactions directly to the issuing bank,
which can then authenticate the cardholder using its preferred method. The
incentive for the merchant is that the transaction is then treated as a cardholderpresent one, so the merchant is no longer at risk. The problem with this is
that the quality of authentication offered by participating banks varies wildly.
At the top of the scale are banks that use two-channel authentication: when
you buy online you get a text message saying something like ‘If you really
want to pay Amazon.com $76.23, enter the code 4697 in your browser now’. At
the bottom end are banks that ask you to enter your ATM PIN into the browser
directly — thereby making their customers wide-open targets for particularly
severe phishing attacks. There is a clear disincentive for the cardholder, who
may now be held liable in many countries regardless of the quality of the local
authentication methods.

10.5 Credit Cards

Of course, even if you have the cardholder physically present, this doesn’t
guarantee that fraud will be rare. For many years, most fraud was done
in person with stolen cards, and stores which got badly hit tended to be
those selling goods that can be easily fenced, such as jewelry and consumer
electronics. Banks responded by lowering their floor limits. More recently, as
technical protection mechanisms have improved, there has been an increase
in scams involving cards that were never received by genuine customers. This
pre-issue fraud can involve thefts from the mail of the many ‘pre-approved’
cards which arrive in junk mail, or even applications made in the names of
people who exist and are creditworthy, but are not aware of the application
(‘identity theft’). These attacks on the system are intrinsically hard to tackle
using purely technical means.

10.5.2

Forgery

In the early 1980’s, electronic terminals were introduced through which a
sales clerk could swipe a card and get an authorization automatically. But
the sales draft was still captured from the embossing, so crooks figured out
how to re-encode the magnetic strip of a stolen card with the account number
and expiry date of a valid card, which they often got by fishing out discarded
receipts from the trash cans of expensive restaurants. A re-encoded card would
authorize perfectly, but when the merchant submitted the draft for payment,
the account number didn’t match the authorization code (a six digit number
typically generated by encrypting the account number, date and amount). So
the merchants didn’t get paid and raised hell.
Banks then introduced terminal draft capture where a sales draft is printed
automatically using the data on the card strip. The crooks’ response was a flood
of forged cards, many produced by Triad gangs: between 1989 and 1992, magnetic strip counterfeiting grew from an occasional nuisance into half the total
fraud losses [7]. VISA’s response was card verification values (CVVs) — these
are three-digit MACs computed on the card strip contents (account number,
version number, expiry date) and written at the end of the strip. They worked
well initially; in the first quarter of 1994, VISA International’s fraud losses
dropped by 15.5% while Mastercard’s rose 67% [269]. So Mastercard adopted
similar checksums too.
The crooks moved to skimming — operating businesses where genuine customer cards were swiped through an extra, unauthorized, terminal to grab
a copy of the magnetic strip, which would then be re-encoded on a genuine card. The banks’ response was intrusion detection systems, which in the
first instance tried to identify criminal businesses by correlating the previous
purchase histories of customers who complained.
In the late 1990’s, credit card fraud rose sharply due to another simple
innovation in criminal technology: the operators of the crooked businesses

345

346

Chapter 10

■

Banking and Bookkeeping

which skim card data absorb the cost of the customer’s transaction rather than
billing it. You have a meal at a Mafia-owned restaurant, offer a card, sign
the voucher, and fail to notice when the charge doesn’t appear on your bill.
Perhaps a year later, there is suddenly a huge bill for jewelry, electrical goods
or even casino chips. By then you’ve completely forgotten about the meal, and
the bank never had a record of it [501].
In the early 2000’s, high-tech criminals became better organised as electronic
crime became specialised. Phishing involved malware writers, botnet herders,
phishing site operators and cash-out specialists, linked by black markets organised in chat rooms. This has spilled over from targeting online transactions to
attacks on retail terminals. Fake terminals, and terminal tapping devices, used
in the USA and Canada simply record mag-strip card and PIN data, which
are used to make card clones for use in ATMs. In the Far East, wiretaps have
been used to harvest card data wholesale [792]. Things are more complex in
Europe which has introduced smartcards, but there are now plenty of devices
that copy the EMV standard smartcards to mag-strip cards that are used in
terminals that accept mag-strip transactions. Some of them use vulnerabilities
in the EMV protocol, and so I’ll come back to discuss them after I’ve described
bank smartcard use in the next section.

10.5.3

Automatic Fraud Detection

There has been a lot of work since the mid-1990s on more sophisticated
financial intrusion detection. Some generic systems do abuse detection using
techniques such as neural networks, but it’s unclear how effective they are.
When fraud is down one year, it’s hailed as a success for the latest fraud
spotting system [101], while when the figures are up a few years later the
vendors let the matter pass quietly [1191].
More convincing are projects undertaken by specific store chains that look
for known patterns of misuse. For example, an electrical goods chain in
the New York area observed that offender profiling (by age, sex, race and
so on) was ineffective, and used purchase profiling instead to cut fraud by
82% in a year. Their technique involved not just being suspicious of high
value purchases, but training staff to be careful when customers were careless
about purchases and spent less than the usual amount of time discussing
options and features. These factors can be monitored online too, but one
important aspect of the New York success is harder for a web site: employee
rewarding. Banks give a $50 reward per bad card captured, which many stores
just keep — so their employees won’t make an effort to spot cards or risk
embarrassment by confronting a customer. In New York, some store staff were
regularly earning a weekly bonus of $150 or more [840].
With online shopping, the only psychology the site designer can leverage is
that of the villain. It has been suggested that an e-commerce site should have

10.5 Credit Cards

an unreasonably expensive ‘platinum’ option that few genuine customers
will want to buy. This performs two functions. First, it helps you to do
basic purchase profiling. Second, it fits with the model of Goldilocks pricing
developed by economists Hal Shapiro and Carl Varian, who point out that the
real effect of airlines’ offering first class fares is to boost sales of business class
seats to travelers who can now convince their bosses (or themselves) that they
are being ‘economical’ [1159]. Another idea is to have a carefully engineered
response to suspect transactions: if you just say ‘bad card, try another one’
then the fraudster probably will. You may even end up being used by the
crooks as an online service that tells them which of their stolen cards are on
the hot list, and this can upset your bank (even though the banks are to blame
for the system design). A better approach is claim that you’re out of stock, so
the bad guy will go elsewhere [1199].
As for electronic banking, it has recently become important to make intrusion
detection systems work better with lower-level mechanisms. A good example
is the real-time man-in-the-middle attack. After banks in the Netherlands
handed out password calculators to their online banking customers, the
response of the phishermen was to phish in real time: the mark would log on
to the phishing site, which would then log on to the bank site and relay the
challenge for the mark to enter into his calculator. The quick fix for this is to
look out for large numbers of logons coming from the same IP address. It’s
likely to be a long struggle, of course; by next year, the bad guys may be using
botnets to host the middleman software.

10.5.4 The Economics of Fraud
There’s a lot of misinformation about credit card fraud, with statistics quoted
selectively to make points. In one beautiful example, VISA was reported to
have claimed that card fraud was up, and that card fraud was down, on the
same day [598].
But a consistent pattern of figures can be dug out of the trade publications.
The actual cost of credit card fraud during the 1990s was about 0.15% of all
international transactions processed by VISA and Mastercard [1087], while
national rates varied from 1% in America through 0.2% in UK to under 0.1%
in France and Spain. The prevailing business culture has a large effect on the
rate. U.S. banks, for example, are much more willing to send out huge junk
mailings of pre-approved cards to increase their customer base, and write off
the inevitable pre-issue fraud as a cost of doing business. In other countries,
banks are more risk-averse.
The case of France is interesting, as it seems at first sight to be an exceptional
case in which a particular technology has brought real benefits. French banks
introduced chip cards for all domestic transactions in the late 1980’s, and this
reduced losses from 0.269% of turnover in 1987 to 0.04% in 1993 and 0.028%

347

348

Chapter 10

■

Banking and Bookkeeping

in 1995. However, there is now an increasing amount of cross-border fraud.
French villains use foreign magnetic stripe cards — particularly from the
UK [498, 1087] — while French chip cards are used at merchants in non-chip
countries [270]. But the biggest reduction in Europe was not in France but in
Spain, where the policy was to reduce all merchant floor limits to zero and
make all transactions online. This cut their losses from 0.21% of turnover in
1988 to 0.008% in 1991 [110].
The lessons appear to be that first, card fraud is cyclical as new defences
are introduced and the villains learn to defeat them; and second, that the most
complicated and expensive technological solution doesn’t necessarily work
best in the field. In fact, villains get smarter all the time. After the UK moved
from magnetic strip cards to chipcards in 2005, it took less than eighteen
months for the crooks to industrialise the process of moving stolen card data
abroad: by 2007, as I’ll discuss shortly.

10.5.5 Online Credit Card Fraud — the Hype
and the Reality
Turning now from traditional credit card fraud to the online variety, I first
helped the police investigate an online credit card fraud in 1987. In that case,
the bad guy got a list of hot credit card numbers from his girlfriend who
worked in a supermarket, and used them to buy software from companies
in California, which he downloaded to order for his customers. This worked
because hot card lists at the time carried only those cards which were being
used fraudulently in that country; it also guaranteed that the bank would
not be able to debit an innocent customer. As it happens, the criminal quit
before there was enough evidence to nail him. A rainstorm washed away the
riverbank opposite his house and exposed a hide which the police had built to
stake him out.
From about 1995, there was great anxiety at the start of the dotcom boom
that the use of credit cards on the Internet would lead to an avalanche of
fraud, as ‘evil hackers’ intercepted emails and web forms and harvested credit
card numbers by the million. These fears drove Microsoft and Netscape to
introduce SSL/TLS to encrypt credit card transactions en route from browsers
to web servers. (There was also a more secure protocol, SET, in which the
browser would get a certificate from the card-issuing bank and would actually
sign the transaction; this failed to take off as the designers didn’t get the
incentives right.)
The hype about risks to credit card numbers was overdone. Intercepting
email is indeed possible but it’s surprisingly difficult in practice — so much
so that governments had to bully ISPs to install snooping devices on their
networks to make court-authorized wiretaps easier [187]. I’ll discuss this
further in Part III. The actual threat is twofold. First, there’s the growth of

10.5 Credit Cards

phishing since 2004; there (as I remarked in Chapter 2) the issue is much more
psychology than cryptography. TLS per se doesn’t help, as the bad guys can
also get certificates and encrypt the traffic.
Second, most of the credit card numbers that are traded online got into bad
hands because someone hacked a merchant’s computer. VISA had had rules
for many years prohibiting merchants from storing credit card data once the
transaction had been processed, but many merchants simply ignored them.
From 2000, VISA added new rules that merchants had to install a firewall,
keep security patches up-to-date, encrypt stored and transmitted data and
regularly update antivirus software [1262]. These were also not enforced. The
latest set of rules, the Payment Card Industry Data Security Standard, are a
joint effort by VISA and Mastercard, and supported by the other brands too;
they say much the same things, and we’ll have to wait and see whether the
enforcement is any better. ‘PCI’, as the new system’s called, certainly seems
to be causing some pain; in October 2007, the U.S. National Retail Federation
asked credit card companies to stop forcing retailers to store credit card data
at all (at present they are supposed to store card numbers temporarily in case
of chargebacks) [1296].
The real incentives facing merchants are, first, the cost of disputes, and
second, the security-breach disclosure laws that are (in 2007) in force in 34 U.S.
states and that are contemplated as a European Directive. Disclosure laws have
had a very definite effect in the USA as the stock prices of companies suffering
a breach can fall several percent. As for disputes, consumer protection laws
in many countries make it easy to repudiate a transaction. Basically all the
customer has to do is call the credit card company and say ‘I didn’t authorize
that’ and the merchant is saddled with the bill. This was workable in the days
when almost all credit card transactions took place locally and most were
for significant amounts. If a customer fraudulently repudiated a transaction,
the merchant would pursue them through the courts and harrass them using
local credit reference agencies. In addition, the banks’ systems are often quite
capable of verifying local cardholder addresses.
But the Internet differs from the old mail order/telephone order regime
in that many transactions are international, amounts are small, and verifying
overseas addresses via the credit card system is problematic. Often all the
call center operator can do is check that the merchant seems confident when
reading an address in the right country. So the opportunity for repudiating
transactions — and getting away with it — is hugely increased. There are
particularly high rates of repudiation of payment to porn sites. No doubt some
of these disputes happen when a transaction made under the influence of a
flush of hormones turns up on the family credit card bill and the cardholder
has to repudiate it to save his marriage; but many are the result of blatant
fraud by operators. A common scam was to offer a ‘free tour’ of the site and
demand a credit card number, supposedly to verify that the user was over 18,

349

350

Chapter 10

■

Banking and Bookkeeping

and then bill him anyway. Some sites billed other consumers who have never
visited them at all [620]. Even apparently large and ‘respectable’ web sites like
playboy.com were criticised for such practices, and at the bottom end of the
porn industry, things are atrocious.
The main brake on wicked websites is the credit-card chargeback. A bank
will typically charge the merchant $100–200 in fees for each of them, as well
as debiting the transaction amount from his account. So if more than a small
percentage of the transactions on your site are challenged by customers, your
margins will be eroded. If chargebacks go over perhaps 10%, your bank may
withdraw your card acquisition service. This has happened to a number of
porn sites; a more prosaic example was the collapse of sportswear merchant
boo.com because they had too many returns: their business model assumed a
no-quibble exchange or refund policy, similar to those operated by high-street
discount clothing stores. Yet more of their shipments than they’d expected
were the wrong size, or the wrong colour, or just didn’t appeal to the customers.
Refunds are cheaper than chargebacks, but still, the credit card penalties broke
the company [1199]. Chargebacks also motivate merchants to take case — to
beware of odd orders (e.g. for four watches), orders from dodgy countries,
customers using free email services, requests for expedited delivery, and so
on. But leaving the bulk of the liability for mail-order transactions with them
is suboptimal: the banks know much more about fraud patterns.
This history suggests that purely technological fixes may not be easy, and
that the most effective controls will be at least partly procedural. Some card
issuers offer credit card numbers that can be used once only; as they issue
them one at a time to customers via their web site, this also helps drive lots of
traffic to their advertisers [324]. Other banks have found that they get better
results by investing in address verification [102]. However the big investment
in the last few years has been in new card technologies, with Europe replacing
both credit cards and debit cards with smartcards complying with the EMV
‘chip and PIN’ standard, while U.S. banks are starting to roll out bank cards
based on RFID.

10.6

Smartcard-Based Banking

In the 1960s and 70s, various people proposed putting integrated circuits
in bank cards. The Germans consider the smartcard to have been invented
by Helmut Gröttrup and Jürgen Dethloff in 1968, when they proposed and
patented putting a custom IC in a card 1968; the French credit Roland Moreno,
who proposed putting memory chips in cards in 1973, and Michel Ugon who
proposed adding a microprocessor in 1977. The French company HoneywellBull patented a chip containing memory, a microcontroller and everything
else needed to do transactions in 1982; they started being used in French

10.6 Smartcard-Based Banking

pay phones in 1983; and in banking from the mid-1980s, as discussed in
section 10.5.4 above.
Smartcards were marketed from the beginning as the French contribution to
the information age, and the nascent industry got huge government subsidies.
In the rest of the world, progress was slower. There were numerous pilot
projects in which smartcards were tried out with different protocols, and in
different applications. I already mentioned the COPAC system at 3.8.1; we
developed this in 1991–2 for use in countries with poor telecommunications,
and it sold best in Russia. Norway’s commercial banks started issuing smartcards in 1986 but its savings banks refused to; when the central bank pressured
the banks to unite on a common technology, mag stripe won and smartcards
were withdrawn in 1995. Britain’s NatWest Bank developed the Mondex electronic purse system in the early 90s, piloted it in Swindon, then sold it to
Mastercard. There was a patent fight between VISA (which had bought the
COPAC rights) and Mastercard. The Belgian banks implemented an electronic
purse called ‘Proton’ for low-value payments to devices like parking meters;
the Germans followed with ‘Geldkarte’ which became the European standard
EN1546 and is now also available as the ‘Moneo’ electronic purse in France.
Offline systems such as Mondex had problems dealing with broken cards.
If the back-end system doesn’t do full balancing, then when a customer
complains that a card has stopped working, all the bank can do is either to
refund the amount the customer claims was on the card, or tell her to get lost;
so most modern systems do balancing, which means they aren’t as cheap to
operate as one might have hoped. All this was good learning experience. But
for a payment card to be truly useful, it has to work internationally — and
especially so in Europe with many small countries jammed up close together,
where even a one-hour shopping trip in the car may involve international
travel. So the banks finally got together with their suppliers and hammered
out a standard.

10.6.1 EMV
The EMV standards are named after the participating institutions Europay,
Mastercard and VISA (Europay developed the Belgian Proton card). As of
2007, several hundred million European cardholders now have debit and
credit cards that conform to this standard, and can be used more or less
interoperably in the UK, Ireland, France, Germany and other participating
countries. In English speaking countries such as the UK and Ireland, EMV
has been branded as ‘chip and PIN’ (although the standards do also support
signature-based transactions). The standards’ proponents hope that they will
become the worldwide norm for card payments, although this is not quite a
done deal: Japan and increasingly the USA are adopting RFID standards for
contactless payment, which I’ll discuss in the next section. Anyway, in much

351

352

Chapter 10

■

Banking and Bookkeeping

of the world, the EMV standards act as a ‘fraud bulldozer’, moving around the
payment-systems landscape so that some types of fraud become less common
and others more so.
The EMV protocol documents [429] are not so much a single protocol as a
suite of protocols. The VISA version of the protocols alone come to more than
3,600 pages, and these are only the compatibility aspects — there are further
documents specific to individual banks. Specifications this complex cannot be
expected to be bug-free, and I’ll describe some of the bugs in the following
sections. The most obvious problem is that the documents allow many options,
some of which are dangerous, either individually or in combination. So EMV
can be thought of as a construction kit for building payment systems, with
which one can build systems that are quite secure, or very insecure, depending
on how various parameters are set, what sort of fallback modes are invoked
on failure, and what other systems are hooked up.
In order to understand this, we need to look briefly at the EMV mechanisms.
Each customer card contains a smartcard chip with the capability to verify
a PIN and authenticate a transaction. The cards come in two types: low-cost
cards that do only symmetric cryptography and use a set of protocols known as
static data authentication (SDA); and more expensive cards that can generate
digital signatures, supporting protocols called dynamic data authentication
(DDA) and combined data authentication (CDA).

10.6.1.1

Static Data Authentication

SDA is the default EMV protocol, and it works as follows. The customer puts
her card into the ‘chip and PIN’ terminal to which it sends a digital certificate,
account number and the other data found on the old-fashioned magnetic strip,
plus a digital signature from the card-issuing bank (the bank chooses which
data items to sign). The terminal verifies the signature and the merchant enters
the payment amount; the terminal solicits the PIN; the customer enters it; and
it’s sent in clear to the card. If the PIN is accepted, the card tells the terminal
that all’s well and generates a MAC, called an ‘application data cryptogram’,
on the supplied data (merchant ID, amount, serial number, nonce and so on).
The key used to compute this MAC is shared between the card and the
customer’s bank, and so it can only be verified by the issuing bank. (The bank
could thus use any algorithm it liked, but the default is DES-CBC-MAC with
triple-DES for the last block.) Also, the only way the terminal can check that
the transaction is genuine is by going online and getting an acknowledgement.
As this isn’t always convenient, some merchants have a ‘floor limit’ below
which offline transactions are permitted.
This protocol has a number of vulnerabilities that are by now well known.
The most commonly-exploited one is backwards compatibility with magnetic
strip cards: as the certificate contains all the information needed to forge a

10.6 Smartcard-Based Banking

mag-strip card, and as the introduction of chip and PIN means that people
now enter PINs everywhere rather than just at ATMs, a number of gangs have
used assorted sniffers to collect card data from terminals and collected money
using mag-strip forgeries. Many ATMs and merchant terminals even in the
EMV adopter countries will fall back to mag-strip processing for reliability
reasons, and there are many countries — from the USA to Thailand — that
haven’t adopted EMV at all. There are two flavours of attack: where the PIN
is harvested along with the card details, and where it’s harvested separately.
First, where the card reader and the PIN pad are separate devices, then a
wiretap between them will get PINs as well as card data. Since 2005 there have
been reports of sniffing devices, made in Eastern Europe, that have been found
in stores in Italy; they harvest the card and PIN data and send it by SMS to the
villains who installed them. This may be done under cover of a false-pretext
‘maintenance’ visit or by corrupt store staff. There have also been reports of
card cloning at petrol stations after PIN pads were replaced with tampered
ones; although these cases are waiting to come to trial as I write, I investigated
the tamper-resistance of PIN pads with two colleagues and we found that the
leading makes were very easy to compromise.
For example, the Ingenico i3300, one of the most widely-deployed terminals
in the UK in 2007, suffers from a series of design flaws. Its rear has a useraccessible compartment, shown in Figure 10.4, for the insertion of optional
extra components. This space is not designed to be tamper-proof, and when
covered it cannot be inspected by the cardholder even if she handles the device.
This compartment gives access to the bottom layer of the circuit board. This
does not give direct access to sensitive data — but, curiously, the designers
opted to provide the attacker 1 mm diameter holes (used for positioning the
optional components) and vias through the circuit board. From there, a simple
metal hook can tap the serial data line. We found that a 1 mm diameter via,
carrying the serial data signal, is easily accessed using a bent paperclip. This can
be inserted through a hole in the plastic surrounding the internal compartment,
and does not leave any external marks. The effect is that the attacker can design
a small wiretap circuit that sits invisibly inside the terminal and gathers both
card and PIN data. This circuit can be powered from the terminal itself and
could contain a small mobile phone to SMS the booty to its makers.
Britain had an epidemic of fraud in 2006–7 apparently involving sniffer
devices inserted into the wiring between card terminals and branch servers in
petrol stations in the UK. As the card readers generally have integral PIN pads
in this application, the PINs may be harvested by eye by petrol-station staff,
many of whom are Tamils who arrived as refugees from the civil war in Sri
Lanka. It’s said that the Tamil Tigers — a terrorist group — intimidates them
into participating. This was discovered when Thai police caught men in Phuket
with 5,000 forged UK debit and credit cards, copied on to ‘white plastic’.

353

354

Chapter 10

■

Banking and Bookkeeping

Figure 10.4: A rigid wire is inserted through a hole in the Ingenico’s concealed compartment wall to intercept the smartcard data. The front of the device is shown on the top
right.

Attacks exploiting the fact that the MAC can’t be read by the merchant
include ‘yescards’. These are cards programmed to accept any PIN (hence the
name) and to participate in the EMV protocol using an externally-supplied
certificate, returning random values for the MAC. A villain with a yescard
and access to genuine card certificates — perhaps through a wiretap on a
merchant terminal — can copy a cert to a yescard, take it to a merchant with
a reasonable floor limit, and do a transaction using any PIN. This attack has
been reported in France and suspected in the UK [122]; it’s pushing France
towards a move from SDA to DDA. However, most such frauds in Britain still
use magnetic strip fallback: many ATMs and merchants use the strip if the
chip is not working.
Another family of problems with EMV has to do with authentication
methods. Each card, and each terminal, has a list of preferred methods, which
might say in effect: ‘first try online PIN verification, and if that’s not supported
use local cleartext PIN verification, and if that’s not possible then you don’t
need to authenticate the customer at all’. It might at first sight be surprising
that ‘no authentication’ is ever an option, but it’s frequently there, in order
to support devices such as parking ticket vending machines that don’t have

10.6 Smartcard-Based Banking

PIN pads. One glitch is that the list of authentication methods isn’t itself
authenticated, so a bad man might manipulate it in a false-terminal or relay
attack. Another possibility is to have two cards: your own card, for which you
know the PIN, and a stolen card for which you don’t, slotted into a device that
lets you switch between them. You present the first card to the terminal and
verify the PIN; you then present the transaction to the stolen card with the
verification method changed to ‘no authentication’. The stolen card computes
the MAC and gets debited. The bank then maintains to the victim that as his
chip was read and a PIN was used, he’s liable for the money.
One countermeasure being contemplated is to insert the verification method
into the transaction data; another is to reprogram cards to remove ‘no authentication’ from the list of acceptable options. If your bank takes the latter option,
you’d better keep some change in your car ashtray! Yet another is to reprogram customer cards so that ‘no authentication’ works only up to some limit,
say $200.
The fact that banks can now reprogram customers’ cards in the field is also
novel. The mechanism uses the shared key material and kicks in when the
card’s at an online terminal such as an ATM. On