OWASP Ing Guide V4
OWASP%20ing%20Guide%20v4
OWASP_ing_Guide_v4
OWASP_ing_Guide_v4
OWASP_ing_Guide_v4
OWASP-ing-Guide-v4
User Manual: Pdf
Open the PDF directly: View PDF 
.
Page Count: 224
| Download | |
| Open PDF In Browser | View PDF |
1
Testing Guide 4.0
Project Leaders: Matteo Meucci and Andrew Muller
Creative Commons (CC) Attribution Share-Alike
Free version at http://www.owasp.org
2
THE ICONS BELOW REPRESENT WHAT
OTHER VERSIONS ARE AVAILABLE IN PRINT
FOR THIS BOOK TITLE.
ALPHA: “Alpha Quality” book content is a
working draft. Content is very rough and in
development until the next level of publishing.
BETA: “Beta Quality” book content is the next
highest level. Content is still in development
until the next publishing.
RELEASE: “Release Quality” book content
is the highest level of quality in a book title’s
lifecycle, and is a final product.
YOU ARE FREE:
To Share - to copy, distribute and
transmit the work
To Remix - to adapt the work
UNDER THE FOLLOWING CONDITIONS:
Attribution. You must attribute the work
in the manner specified by the author or
licensor (but not in any way that suggests
that they endorse you or your use of the
work).
Share Alike. If you alter, transform, or
build upon this work, you may distribute
the resulting work only under the same,
similar or a compatible license.
ALPHA
BETA
RELEASE
The Open Web Application Security Project (OWASP) is a worldwide free and open community focused on improving the security of application software. Our mission is to make
application security “visible”, so that people and organizations can make informed decisions
about application security risks. Every one is free to participate in OWASP and all of our
materials are available under a free and open software license. The OWASP Foundation
is a 501c3 not-for-profit charitable organization that ensures the ongoing availability and
support for our work.
Testing Guide Foreword - Table of contents
3-4
5-6
0
Foreword by Eoin Keary
1
Frontispiece
About the OWASP Testing Guide Project
About The Open Web Application Security Project
7 - 21
2
Introduction
The OWASP Testing Project
Principles of Testing
Testing Techniques Explained
Deriving Security Test Requirements
Security Tests Integrated in Development and Testing Workflows
Security Test Data Analysis and Reporting
22 - 24
3
The OWASP Testing Framework
Overview
Phase 1: Before Development Begins
Phase 2: During Definition and Design
Phase 3: During Development
Phase 4: During Deployment
Phase 5: Maintenance and Operations
A Typical SDLC Testing Workflow
25 - 207
4
Web Application Security Testing
Introduction and Objectives
Testing Checklist
Information Gathering
Conduct Search Engine Discovery and Reconnaissance for Information Leakage (OTG-INFO-001)
Fingerprint Web Server (OTG-INFO-002)
Review Webserver Metafiles for Information Leakage (OTG-INFO-003)
Enumerate Applications on Webserver (OTG-INFO-004)
Review Webpage Comments and Metadata for Information Leakage (OTG-INFO-005)
Identify application entry points (OTG-INFO-006)
Map execution paths through application (OTG-INFO-007)
Fingerprint Web Application Framework (OTG-INFO-008)
Fingerprint Web Application (OTG-INFO-009)
Map Application Architecture (OTG-INFO-010)
Configuration and Deployment Management Testing
Test Network/Infrastructure Configuration (OTG-CONFIG-001)
Test Application Platform Configuration (OTG-CONFIG-002)
Testing Guide Foreword - Table of contents
Test File Extensions Handling for Sensitive Information (OTG-CONFIG-003)
Review Old, Backup and Unreferenced Files for Sensitive Information (OTG-CONFIG-004)
Enumerate Infrastructure and Application Admin Interfaces (OTG-CONFIG-005)
Test HTTP Methods (OTG-CONFIG-006)
Test HTTP Strict Transport Security (OTG-CONFIG-007)
Test RIA cross domain policy (OTG-CONFIG-008)
Identity Management Testing
Test Role Definitions (OTG-IDENT-001)
Test User Registration Process (OTG-IDENT-002)
Test Account Provisioning Process (OTG-IDENT-003)
Testing for Account Enumeration and Guessable User Account (OTG-IDENT-004)
Testing for Weak or unenforced username policy (OTG-IDENT-005)
Authentication Testing
Testing for Credentials Transported over an Encrypted Channel (OTG-AUTHN-001)
Testing for default credentials (OTG-AUTHN-002)
Testing for Weak lock out mechanism (OTG-AUTHN-003)
Testing for bypassing authentication schema (OTG-AUTHN-004)
Test remember password functionality (OTG-AUTHN-005)
Testing for Browser cache weakness (OTG-AUTHN-006)
Testing for Weak password policy (OTG-AUTHN-007)
Testing for Weak security question/answer (OTG-AUTHN-008)
Testing for weak password change or reset functionalities (OTG-AUTHN-009)
Testing for Weaker authentication in alternative channel (OTG-AUTHN-010)
Authorization Testing
Testing Directory traversal/file include (OTG-AUTHZ-001)
Testing for bypassing authorization schema (OTG-AUTHZ-002)
Testing for Privilege Escalation (OTG-AUTHZ-003)
Testing for Insecure Direct Object References (OTG-AUTHZ-004)
Session Management Testing
Testing for Bypassing Session Management Schema (OTG-SESS-001)
Testing for Cookies attributes (OTG-SESS-002)
Testing for Session Fixation (OTG-SESS-003)
Testing for Exposed Session Variables (OTG-SESS-004)
Testing for Cross Site Request Forgery (CSRF) (OTG-SESS-005)
Testing for logout functionality (OTG-SESS-006)
Test Session Timeout (OTG-SESS-007)
Testing for Session puzzling (OTG-SESS-008)
Input Validation Testing
Testing for Reflected Cross Site Scripting (OTG-INPVAL-001)
Testing for Stored Cross Site Scripting (OTG-INPVAL-002)
Testing for HTTP Verb Tampering (OTG-INPVAL-003)
Testing for HTTP Parameter pollution (OTG-INPVAL-004)
Testing for SQL Injection (OTG-INPVAL-005)
Oracle Testing
MySQL Testing
SQL Server Testing
Testing PostgreSQL (from OWASP BSP)
MS Access Testing
3
Testing Guide Foreword - Table of contents
Testing for NoSQL injection
Testing for LDAP Injection (OTG-INPVAL-006)
Testing for ORM Injection (OTG-INPVAL-007)
Testing for XML Injection (OTG-INPVAL-008)
Testing for SSI Injection (OTG-INPVAL-009)
Testing for XPath Injection (OTG-INPVAL-010)
IMAP/SMTP Injection (OTG-INPVAL-011)
Testing for Code Injection (OTG-INPVAL-012)
Testing for Local File Inclusion
Testing for Remote File Inclusion
Testing for Command Injection (OTG-INPVAL-013)
Testing for Buffer overflow (OTG-INPVAL-014)
Testing for Heap overflow
Testing for Stack overflow
Testing for Format string
Testing for incubated vulnerabilities (OTG-INPVAL-015)
Testing for HTTP Splitting/Smuggling (OTG-INPVAL-016)
Testing for Error Handling
Analysis of Error Codes (OTG-ERR-001)
Analysis of Stack Traces (OTG-ERR-002)
Testing for weak Cryptography
Testing for Weak SSL/TLS Ciphers, Insufficient Transport Layer Protection (OTG-CRYPST-001)
Testing for Padding Oracle (OTG-CRYPST-002)
Testing for Sensitive information sent via unencrypted channels (OTG-CRYPST-003)
Business Logic Testing
Test Business Logic Data Validation (OTG-BUSLOGIC-001)
Test Ability to Forge Requests (OTG-BUSLOGIC-002)
Test Integrity Checks (OTG-BUSLOGIC-003)
Test for Process Timing (OTG-BUSLOGIC-004)
Test Number of Times a Function Can be Used Limits (OTG-BUSLOGIC-005)
Testing for the Circumvention of Work Flows (OTG-BUSLOGIC-006)
Test Defenses Against Application Mis-use (OTG-BUSLOGIC-007)
Test Upload of Unexpected File Types (OTG-BUSLOGIC-008)
Test Upload of Malicious Files (OTG-BUSLOGIC-009)
Client Side Testing
Testing for DOM based Cross Site Scripting (OTG-CLIENT-001)
Testing for JavaScript Execution (OTG-CLIENT-002)
Testing for HTML Injection (OTG-CLIENT-003)
Testing for Client Side URL Redirect (OTG-CLIENT-004)
Testing for CSS Injection (OTG-CLIENT-005)
Testing for Client Side Resource Manipulation (OTG-CLIENT-006)
Test Cross Origin Resource Sharing (OTG-CLIENT-007)
Testing for Cross Site Flashing (OTG-CLIENT-008)
Testing for Clickjacking (OTG-CLIENT-009)
Testing WebSockets (OTG-CLIENT-010)
Test Web Messaging (OTG-CLIENT-011)
Test Local Storage (OTG-CLIENT-012)
4
Testing Guide Foreword - Table of contents
208 - 222
5
Reporting
Appendix A: Testing Tools
Black Box Testing Tools
Appendix B: Suggested Reading
Whitepapers
Books
Useful Websites
Appendix C: Fuzz Vectors
Fuzz Categories
Appendix D: Encoded Injection
Input Encoding
Output Encoding
5
Testing Guide Foreword - By Eoin Keary
0
Testing Guide Foreword
The problem of insecure software is perhaps the
most important technical challenge of our time. The
dramatic rise of web applications enabling business,
social networking etc has only compounded the
requirements to establish a robust approach to writing
and securing our Internet, Web Applications and Data.
Foreword by Eoin Keary, OWASP Global Board
The problem of insecure software is perhaps the most important
technical challenge of our time. The dramatic rise of web applications enabling business, social networking etc has only compounded the requirements to establish a robust approach to writing and securing our Internet, Web Applications and Data.
At The Open Web Application Security Project (OWASP), we’re
trying to make the world a place where insecure software is the
anomaly, not the norm. The OWASP Testing Guide has an important role to play in solving this serious issue. It is vitally important
that our approach to testing software for security issues is based
on the principles of engineering and science. We need a consistent, repeatable and defined approach to testing web applications.
A world without some minimal standards in terms of engineering
and technology is a world in chaos.
It goes without saying that you can’t build a secure application
without performing security testing on it. Testing is part of a wider
approach to building a secure system. Many software development organizations do not include security testing as part of their
standard software development process. What is even worse is
that many security vendors deliver testing with varying degrees
of quality and rigor.
Security testing, by itself, isn’t a particularly good stand alone
measure of how secure an application is, because there are an infinite number of ways that an attacker might be able to make an
application break, and it simply isn’t possible to test them all. We
can’t hack ourselves secure and we only have a limited time to test
and defend where an attacker does not have such constraints.
In conjunction with other OWASP projects such as the Code review
Guide, the Development Guide and tools such as OWASP ZAP, this
is a great start towards building and maintaining secure applications. The Development Guide will show your project how to architect and build a secure application, the Code Review Guide will tell
you how to verify the security of your application’s source code,
and this Testing Guide will show you how to verify the security of
your running application. I highly recommend using these guides
as part of your application security initiatives.
Why OWASP?
Creating a guide like this is a huge undertaking, requiring the expertise of hundreds of people around the world. There are many
different ways to test for security flaws and this guide captures
the consensus of the leading experts on how to perform this testing quickly, accurately, and efficiently. OWASP gives like minded
security folks the ability to work together and form a leading practice approach to a security problem.
The importance of having this guide available in a completely free
and open way is important for the foundations mission. It gives
anyone the ability to understand the techniques used to test for
common security issues. Security should not be a black art or
closed secret that only a few can practice. It should be open to all
and not exclusive to security practitioners but also QA, Developers
6
Testing Guide Foreword - By Eoin Keary
and Technical Managers. The project to build this guide keeps this
expertise in the hands of the people who need it - you, me and
anyone that is involved in building software.
This guide must make its way into the hands of developers and
software testers. There are not nearly enough application security
experts in the world to make any significant dent in the overall
problem. The initial responsibility for application security must
fall on the shoulders of the developers, they write the code. It
shouldn’t be a surprise that developers aren’t producing secure
code if they’re not testing for it or consider the types of bugs
which introduce vulnerability.
Keeping this information up to date is a critical aspect of this guide
project. By adopting the wiki approach, the OWASP community
can evolve and expand the information in this guide to keep pace
with the fast moving application security threat landscape.
This Guide is a great testament to the passion and energy our
members and project volunteers have for this subject. It shall certainly help change the world a line of code at a time.
Tailoring and Prioritizing
You should adopt this guide in your organization. You may need to
tailor the information to match your organization’s technologies,
processes, and organizational structure.
In general there are several different roles within organizations
that may use this guide:
• Developers should use this guide to ensure that they are producing secure code. These tests should be a part of normal code and
unit testing procedures.
• Software testers and QA should use this guide to expand the set
of test cases they apply to applications. Catching these vulnerabilities early saves considerable time and effort later.
• Security specialists should use this guide in combination with
other techniques as one way to verify that no security holes have
been missed in an application.
• Project Managers should consider the reason this guide exists
and that security issues are manifested via bugs in code and design.
The most important thing to remember when performing security
testing is to continuously re-prioritize. There are an infinite number of possible ways that an application could fail, and organizations always have limited testing time and resources. Be sure time
and resources are spent wisely. Try to focus on the security holes
that are a real risk to your business. Try to contextualize risk in
terms of the application and its use cases.
This guide is best viewed as a set of techniques that you can use
to find different types of security holes. But not all the techniques
are equally important. Try to avoid using the guide as a checklist,
new vulnerabilities are always manifesting and no guide can be
an exhaustive list of “things to test for”, but rather a great place
to start.
The Role of Automated Tools
There are a number of companies selling automated security analysis and testing tools. Remember the limitations of these tools
so that you can use them for what they’re good at. As Michael
Howard put it at the 2006 OWASP AppSec Conference in Seattle,
“Tools do not make software secure! They help scale the process
and help enforce policy.”
Most importantly, these tools are generic - meaning that they are
not designed for your custom code, but for applications in general.
That means that while they can find some generic problems, they
do not have enough knowledge of your application to allow them
to detect most flaws. In my experience, the most serious security
issues are the ones that are not generic, but deeply intertwined in
your business logic and custom application design.
These tools can also be seductive, since they do find lots of potential issues. While running the tools doesn’t take much time, each
one of the potential problems takes time to investigate and verify. If the goal is to find and eliminate the most serious flaws as
quickly as possible, consider whether your time is best spent with
automated tools or with the techniques described in this guide.
Still, these tools are certainly part of a well-balanced application
security program. Used wisely, they can support your overall processes to produce more secure code.
Call to Action
If you’re building, designing or testing software, I strongly encourage you to get familiar with the security testing guidance in this
document. It is a great road map for testing the most common
issues facing applications today, but it is not exhaustive. If you
find errors, please add a note to the discussion page or make the
change yourself. You’ll be helping thousands of others who use
this guide.
Please consider joining us as an individual or corporate member so
that we can continue to produce materials like this testing guide
and all the other great projects at OWASP.
Thank you to all the past and future contributors to this guide,
your work will help to make applications worldwide more secure.
Eoin Keary, OWASP Board Member, April 19, 2013
7
Testing Guide Frontispiece
1
Testing Guide Frontispiece
“Open and collaborative knowledge: that is the
OWASP way.”
With V4 we realized a new guide that will be the
standard de-facto guide to perform Web Application
Penetration Testing
“Open and collaborative knowledge: that is the OWASP way.”
With V4 we realized a new guide that will be the standard de-facto guide to perform Web Application Penetration Testing. - Matteo
Meucci
OWASP thanks the many authors, reviewers, and editors for their
hard work in bringing this guide to where it is today. If you have any
comments or suggestions on the Testing Guide, please e-mail the
Testing Guide mail list:
http://lists.owasp.org/mailman/listinfo/owasp-testing
Or drop an e-mail to the project leaders: Andrew Muller and Matteo Meucci
Revision History
The Testing Guide v4 will be released in 2014. The Testing guide originated in 2003 with Dan Cuthbert as one of the original editors. It was
handed over to Eoin Keary in 2005 and transformed into a wiki. Matteo Meucci has taken on the Testing guide and is now the lead of the
OWASP Testing Guide Project. From 2012 Andrew Muller co-leadership the project with Matteo Meucci.
2014
• “OWASP Testing Guide”, Version 4.0
15th September, 2008
• “OWASP Testing Guide”, Version 3.0
December 25, 2006
• “OWASP Testing Guide”, Version 2.0
Version 4.0
July 14, 2004
• “OWASP Web Application Penetration Checklist”, Version 1.1
[1] This version of the Testing Guide integrates with the two other
flagship OWASP documentation products: the Developers Guide and
the Code Review Guide. To achieve this we aligned the testing categories and test numbering with those in other OWASP products. The
aim of the Testing and Code Review Guides is to evaluate the security
controls described by the Developers Guide.
December 2004
• “The OWASP Testing Guide”, Version 1.0
The OWASP Testing Guide version 4 improves on version 3 in three ways:
Project Leaders
[2] All chapters have been improved and test cases expanded to 87
(64 test cases in v3) including the introduction of four new chapters
and controls:
• Identity Management Testing
• Error Handling
• Cryptography
• Client Side Testing
[3] This version of the Testing Guide encourages the community not
to simply accept the test cases outlined in this guide. We encourage
security testers to integrate with other software testers and devise
test cases specific to the target application. As we find test cases that
have wider applicability we encourage the security testing community
to share them and contribute them to the Testing Guide. This will continue to build the application security body of knowledge and allow
the development of the Testing Guide to be an iterative rather than
monolithic process.
Copyright and License
Copyright (c) 2014 The OWASP Foundation.
This document is released under the Creative Commons 2.5 License.
Please read and understand the license and copyright conditions.
Matteo Meucci
Andrew Muller
Andrew Muller: OWASP Testing Guide Lead since 2013.
Matteo Meucci: OWASP Testing Guide Lead since 2007.
Eoin Keary: OWASP Testing Guide 2005-2007 Lead.
Daniel Cuthbert: OWASP Testing Guide 2003-2005 Lead.
8
Testing Guide Frontispiece
v4 Authors
• Matteo Meucci
• Pavol Luptak
• Marco Morana
• Giorgio Fedon
• Stefano Di Paola
• Gianrico Ingrosso
• Giuseppe Bonfà
• Andrew Muller
• Robert Winkel
• Roberto Suggi Liverani
• Robert Smith
• Tripurari Rai
v4 Reviewers
• Davide Danelon
• Andrea Rosignoli
• Irene Abezgauz
• Lode Vanstechelman
• Sebastien Gioria
• Yiannis Pavlosoglou
• Aditya Balapure
v2 Authors
• Vicente Aguilera
• Mauro Bregolin
• Tom Brennan
• Gary Burns
• Luca Carettoni
• Dan Cornell
• Mark Curphey
• Daniel Cuthbert
• Sebastien Deleersnyder
• Stephen DeVries
v2 Reviewers
• Vicente Aguilera
• Marco Belotti
• Mauro Bregolin
• Marco Cova
• Daniel Cuthbert
• Paul Davies
• Stefano Di Paola
• Matteo G.P. Flora
• Simona Forti
• Darrell Groundy
• Thomas Ryan
• Tim Bertels
• Cecil Su
• Aung KhAnt
• Norbert Szetei
• Michael Boman
• Wagner Elias
• Kevin Horvat
• Tom Brennan
• Tomas Zatko
• Juan Galiana Lara
• Sumit Siddharth
• Mike Hryekewicz
• Simon Bennetts
• Ray Schippers
• Raul Siles
• Jayanta Karmakar
• Brad Causey
• Vicente Aguilera
• Ismael Gonçalves
• David Fern
• Tom Eston
• Kevin Horvath
• Rick Mitchell
v3 Authors
• Anurag Agarwwal
• Daniele Bellucci
• Ariel Coronel
• Stefano Di Paola
• Giorgio Fedon
• Adam Goodman
• Christian Heinrich
• Kevin Horvath
• Gianrico Ingrosso
• Roberto Suggi Liverani
• Kuza55
• Pavol Luptak
• Ferruh Mavituna
• Marco Mella
• Matteo Meucci
• Marco Morana
• Antonio Parata
• Cecil Su
• Harish Skanda Sureddy
• Mark Roxberry
• Andrew Van der Stock
• Stefano Di Paola
• David Endler
• Giorgio Fedon
• Javier Fernández-Sanguino
• Glyn Geoghegan
• Stan Guzik
• Madhura Halasgikar
• Eoin Keary
• David Litchfield
• Andrea Lombardini
• Eoin Keary
• James Kist
• Katie McDowell
• Marco Mella
• Matteo Meucci
• Syed Mohamed
• Antonio Parata
• Alberto Revelli
• Mark Roxberry
• Dave Wichers
• Ralph M. Los
• Claudio Merloni
• Matteo Meucci
• Marco Morana
• Laura Nunez
• Gunter Ollmann
• Antonio Parata
• Yiannis Pavlosoglou
• Carlo Pelliccioni
• Harinath Pudipeddi
• Eduardo Castellanos
• Simone Onofri
• Harword Sheen
• Amro AlOlaqi
• Suhas Desai
• Ryan Dewhurst
• Zaki Akhmad
• Davide Danelon
• Alexander Antukh
• Thomas Kalamaris
• Alexander Vavousis
• Christian Heinrich
• Babu Arokiadas
• Rob Barnes
• Ben Walther
• Anant Shrivastava
• Colin Watson
• Luca Carettoni
• Eoin Keary
• Jeff Williams
• Juan Manuel Bahamonde
• Thomas Skora
• Irene Abezgauz
• Hugo Costa
v3 Reviewers
• Marco Cova
• Kevin Fuller
• Matteo Meucci
• Nam Nguyen
• Rick Mitchell
• Alberto Revelli
• Mark Roxberry
• Tom Ryan
• Anush Shetty
• Larry Shields
• Dafydd Studdard
• Andrew van der Stock
• Ariel Waissbein
• Jeff Williams
• Tushar Vartak
Trademarks
• Java, Java Web Server, and JSP are registered trademarks
of Sun Microsystems, Inc.
• Merriam-Webster is a trademark of Merriam-Webster, Inc.
• Microsoft is a registered trademark of Microsoft Corporation.
• Octave is a service mark of Carnegie Mellon University.
• VeriSign and Thawte are registered trademarks
of VeriSign, Inc.
• Visa is a registered trademark of VISA USA.
• OWASP is a registered trademark of the OWASP Foundation
All other products and company names may be trademarks of their
respective owners. Use of a term in this document should not be
regarded as affecting the validity of any trademark or service mark.
9
Testing Guide Introduction
2
The OWASP Testing Project
The OWASP Testing Project has been in development
for many years. The aim of the project is to help people
understand the what, why, when, where, and how of
testing web applications.
Writing the Testing Guide has proven to be a difficult task. It was a
challenge to obtain consensus and develop content that allowed people to apply the concepts described in the guide, while also enabling
them to work in their own environment and culture. It was also a challenge to change the focus of web application testing from penetration
testing to testing integrated in the software development life cycle.
However, the group is very satisfied with the results of the project.
Many industry experts and security professionals, some of whom are
responsible for software security at some of the largest companies in
the world, are validating the testing framework. This framework helps
organizations test their web applications in order to build reliable and
secure software. The framework does not simply highlighting areas
of weakness, although the latter is certainly a by product of many of
the OWASP guides and checklists. As such, hard decisions had top
be made about the appropriateness of certain testing techniques
and technologies. The group fully understands that not everyone will
agree upon all of these decisions. However, OWASP is able to take the
high ground and change culture over time through awareness and education based on consensus and experience.
The rest of this guide is organized as follows: This introduction covers the pre-requisites of testing web applications and the scope of
testing. It also covers the principles of successful testing and testing
techniques. Chapter 3 presents the OWASP Testing Framework and
explains its techniques and tasks in relation to the various phases of
the software development life cycle. Chapter 4 covers how to test for
specific vulnerabilities (e.g., SQL Injection) by code inspection and penetration testing.
Measuring Security: the Economics of Insecure Software
A basic tenet of software engineering is that you can’t control what
you can’t measure [1]. Security testing is no different. Unfortunately,
measuring security is a notoriously difficult process. This topic will not
be covered in detail here, as it would take a guide on its own (for an
introduction, see [2]).
One aspect that should be emphasized is that security measurements are about both the specific technical issues (e.g., how prevalent
a certain vulnerability is) and how these issues affect the economics
of software. Most technical people will at least understand the basic
issues, or they may have a deeper understanding of the vulnerabilities.
Sadly, few are able to translate that technical knowledge into monetary terms and quantify the potential cost of vulnerabilities to the application owner’s business. Until this happens, CIOs will not be able to
develop an accurate return on security investment and, subsequently,
assign appropriate budgets for software security.
While estimating the cost of insecure software may appear a daunting task, there has been a significant amount of work in this direction.
11
For example, in June 2002, the US National Institute of Standards
(NIST) published a survey on the cost of insecure software to the US
economy due to inadequate software testing [3]. Interestingly, they
estimate that a better testing infrastructure would save more than a
third of these costs, or about $22 billion a year. More recently, the links
between economics and security have been studied by academic researchers. See [4] for more information about some of these efforts.
While estimating the cost of insecure software may appear a daunting task, there has been a significant amount of work in this direction.
For example, in June 2002, the US National Institute of Standards
(NIST) published a survey on the cost of insecure software to the US
economy due to inadequate software testing [3]. Interestingly, they
estimate that a better testing infrastructure would save more than a
third of these costs, or about $22 billion a year. More recently, the links
between economics and security have been studied by academic researchers. See [4] for more information about some of these efforts.
The framework described in this document encourages people to
measure security throughout the entire development process. They
can then relate the cost of insecure software to the impact it has on
the business, and consequently develop appropriate business processes and assign resources to manage the risk. Remember that
measuring and testing web applications is even more critical than for
other software, since web applications are exposed to millions of users through the Internet.
What is Testing?
During the development life cycle of a web application many things
need to be tested, but what does testing actually mean? The Merriam-Webster Dictionary describes testing as:
• To put to test or proof.
• To undergo a test.
• To be assigned a standing or evaluation based on tests.
For the purposes of this document testing is a process of comparing
the state of a system or application against a set of criteria. In the security industry people frequently test against a set of mental criteria
that are neither well defined nor complete. As a result of this, many
outsiders regard security testing as a black art. The aim of this document is to change that perception and to make it easier for people
without in-depth security knowledge to make a difference in testing.
Why Perform Testing?
This document is designed to help organizations understand what
comprises a testing program, and to help them identify the steps that
need to be undertaken to build and operate a testing program on web
applications. The guide gives a broad view of the elements required to
10
Testing Guide Introduction
make a comprehensive web application security program. This guide
can be used as a reference guide and as a methodology to help determine the gap between existing practices and industry best practices.
This guide allows organizations to compare themselves against industry peers, to understand the magnitude of resources required to test
and maintain software, or to prepare for an audit. This chapter does
not go into the technical details of how to test an application, as the
intent is to provide a typical security organizational framework. The
technical details about how to test an application, as part of a penetration test or code review, will be covered in the remaining parts of
this document.
When to Test?
Most people today don’t test software until it has already been created
and is in the deployment phase of its life cycle (i.e., code has been created and instantiated into a working web application). This is generally
a very ineffective and cost-prohibitive practice. One of the best methods to prevent security bugs from appearing in production applications
is to improve the Software Development Life Cycle (SDLC) by including
security in each of its phases. An SDLC is a structure imposed on the
development of software artefacts. If an SDLC is not currently being
used in your environment, it is time to pick one! The following figure
shows a generic SDLC model as well as the (estimated) increasing cost
of fixing security bugs in such a model.
DE
SIG
NE
FI
DEV
ELOP
INTAIN
A
M
N
DE
Figure 1: Generic SDLC Model
DEPLOY
Companies should inspect their overall SDLC to ensure that security
is an integral part of the development process. SDLCs should include
security tests to ensure security is adequately covered and controls
are effective throughout the development process.
What to Test?
It can be helpful to think of software development as a combination of
people, process, and technology. If these are the factors that “create”
software, then it is logical that these are the factors that must be test-
ed. Today most people generally test the technology or the software
itself.
An effective testing program should have components that test:
People – to ensure that there is adequate education and awareness;
Process – to ensure that there are adequate policies and standards
and that people know how to follow these policies;
Technology – to ensure that the process has been effective in its implementation.
Unless a holistic approach is adopted, testing just the technical implementation of an application will not uncover management or operational vulnerabilities that could be present. By testing the people, policies, and processes, an organization can catch issues that would later
manifest themselves into defects in the technology, thus eradicating
bugs early and identifying the root causes of defects. Likewise, testing
only some of the technical issues that can be present in a system will
result in an incomplete and inaccurate security posture assessment.
Denis Verdon, Head of Information Security at Fidelity National Financial presented an excellent analogy for this misconception at the
OWASP AppSec 2004 Conference in New York [5]: “If cars were built
like applications [...] safety tests would assume frontal impact only.
Cars would not be roll tested, or tested for stability in emergency maneuvers, brake effectiveness, side impact, and resistance to theft.”
Feedback and Comments
As with all OWASP projects, we welcome comments and feedback.
We especially like to know that our work is being used and that it is
effective and accurate.
There are some common misconceptions when developing a testing
methodology to find security bugs in software. This chapter covers
some of the basic principles that professionals should take into account when performing security tests on software.
Principles of Testing
There is No Silver Bullet
While it is tempting to think that a security scanner or application
firewall will provide many defenses against attack or identify a multitude of problems, in reality there is no silver bullet to the problem
of insecure software. Application security assessment software, while
useful as a first pass to find low-hanging fruit, is generally immature
and ineffective at in-depth assessments or providing adequate test
coverage. Remember that security is a process and not a product.
Think Strategically, Not Tactically
Over the last few years, security professionals have come to realize
the fallacy of the patch-and-penetrate model that was pervasive in
information security during the 1990’s. The patch-and-penetrate
model involves fixing a reported bug, but without proper investigation
of the root cause. This model is usually associated with the window of
vulnerability shown in the figure below. The evolution of vulnerabilities
in common software used worldwide has shown the ineffectiveness
of this model. For more information about the window of vulnerability
please refer to [6].
Vulnerability studies [7] have shown that with the reaction time of
attackers worldwide, the typical window of vulnerability does not pro12
11
Testing Guide Introduction
vide enough time for patch installation, since the time between a vulnerability being uncovered and an automated attack against it being
developed and released is decreasing every year.
There are several incorrect assumptions in the patch-and-penetrate
model. Many users believe that patches interfere with normal operations and might break existing applications. It is also incorrect to
assume that all users are aware of newly released patches. Consequently not all users of a product will apply patches, either because
they think patching may interfere with how the software works or be-
phases may change depending on the SDLC model used by an organization, each conceptual phase of the archetype SDLC will be used to
develop the application (i.e., define, design, develop, deploy, maintain).
Each phase has security considerations that should become part of
the existing process, to ensure a cost-effective and comprehensive
security program.
There are several secure SDLC frameworks that exist that provide
both descriptive and prescriptive advice. Whether a person takes descriptive or prescriptive advice depends on the maturity of the SDLC
Figure 2: Window of Vulnerability
Vulerability is know
to the vendor
A security vulerability
is discovered
The vendor
notifies it’s clients
(sometimes)
Vulerability is
made pubic
Risk
Level
Securtity
tools are
udpdated (IDS
signatures,
new modules
for VA tools)
A patch is
published
The existence
of the patch is
widely known
The patch is
installed in
all systems
affected
Time
cause they lack knowledge about the existence of the patch.
It is essential to build security into the Software Development Life
Cycle (SDLC) to prevent reoccurring security problems within an application. Developers can build security into the SDLC by developing
standards, policies, and guidelines that fit and work within the development methodology. Threat modeling and other techniques should
be used to help assign appropriate resources to those parts of a system that are most at risk.
The SDLC is King
The SDLC is a process that is well-known to developers. By integrating
security into each phase of the SDLC, it allows for a holistic approach
to application security that leverages the procedures already in place
within the organization. Be aware that while the names of the various
process. Essentially, prescriptive advice shows how the secure SDLC
should work, and descriptive advice shows how its used in the real
world. Both have their place. For example, if you don’t know where
to start, a prescriptive framework can provide a menu of potential
security controls that can be applied within the SDLC. Descriptive advice can then help drive the decision process by presenting what has
worked well for other organizations. Descriptive secure SDLCs include
BSIMM-V; and the prescriptive secure SDLCs inculde OWASP’s Open
Software Assurance Maturity Model (OpenSAMM) and ISO/IEC 27034
Parts 1-8, parts of which are still in development.
Test Early and Test Often
When a bug is detected early within the SDLC it can be addressed faster and at a lower cost. A security bug is no different from a functional
12
Testing Guide Introduction
or performance-based bug in this regard. A key step in making this
possible is to educate the development and QA teams about common
security issues and the ways to detect and prevent them. Although
new libraries, tools, or languages can help design better programs
(with fewer security bugs), new threats arise constantly and developers must be aware of the threats that affect the software they are
developing. Education in security testing also helps developers acquire
the appropriate mindset to test an application from an attacker’s perspective. This allows each organization to consider security issues as
part of their existing responsibilities.
Understand the Scope of Security
It is important to know how much security a given project will require. The information and assets that are to be protected should
be given a classification that states how they are to be handled (e.g.,
confidential, secret, top secret). Discussions should occur with legal
council to ensure that any specific security requirements will be met.
In the USA requirements might come from federal regulations, such
as the Gramm-Leach-Bliley Act [8], or from state laws, such as the
California SB-1386 [9]. For organizations based in EU countries, both
country-specific regulation and EU Directives may apply. For example,
Directive 96/46/EC4 [10] makes it mandatory to treat personal data
in applications with due care, whatever the application.
Develop the Right Mindset
Successfully testing an application for security vulnerabilities requires
thinking “outside of the box.” Normal use cases will test the normal
behavior of the application when a user is using it in the manner that is
expected. Good security testing requires going beyond what is expected and thinking like an attacker who is trying to break the application.
Creative thinking can help to determine what unexpected data may
cause an application to fail in an insecure manner. It can also help find
what assumptions made by web developers are not always true and
how they can be subverted. One of the reasons why automated tools
are actually bad at automatically testing for vulnerabilities is that this
creative thinking must be done on a case-by-case basis as most web
applications are being developed in a unique way (even when using
common frameworks).
Understand the Subject
One of the first major initiatives in any good security program should
be to require accurate documentation of the application. The architecture, data-flow diagrams, use cases, etc, should be written in formal
documents and made available for review. The technical specification
and application documents should include information that lists not
only the desired use cases, but also any specifically disallowed use
case. Finally, it is good to have at least a basic security infrastructure
that allows the monitoring and trending of attacks against an organization’s applications and network (e.g., IDS systems).
Use the Right Tools
While we have already stated that there is no silver bullet tool, tools
do play a critical role in the overall security program. There is a range
of open source and commercial tools that can automate many routine security tasks. These tools can simplify and speed up the security
process by assisting security personnel in their tasks. However, it is
important to understand exactly what these tools can and cannot do
so that they are not oversold or used incorrectly.
The Devil is in the Details
It is critical not to perform a superficial security review of an applica-
tion and consider it complete. This will instill a false sense of confidence that can be as dangerous as not having done a security review
in the first place. It is vital to carefully review the findings and weed out
any false positive that may remain in the report. Reporting an incorrect
security finding can often undermine the valid message of the rest of
a security report. Care should be taken to verify that every possible
section of application logic has been tested, and that every use case
scenario was explored for possible vulnerabilities.
Use Source Code When Available
While black box penetration test results can be impressive and useful
to demonstrate how vulnerabilities are exposed in a production environment, they are not the most effective or efficient way to secure
an application. It is difficult for dynamic testing to test the entire code
base, particularly if many nested conditional statements exist. If the
source code for the application is available, it should be given to the
security staff to assist them while performing their review. It is possible to discover vulnerabilities within the application source that would
be missed during a black box engagement.
Develop Metrics
An important part of a good security program is the ability to determine if things are getting better. It is important to track the results of
testing engagements, and develop metrics that will reveal the application security trends within the organization.
Good metrics will show:
• If more education and training are required;
• If there is a particular security mechanism that is not clearly
understood by the development team;
• If the total number of security related problems being found
each month is going down.
Consistent metrics that can be generated in an automated way from
available source code will also help the organization in assessing the
effectiveness of mechanisms introduced to reduce security bugs in
software development. Metrics are not easily developed, so using
standard metrics like those provided by the OWASP Metrics project
and other organizations is a good starting point.
Document the Test Results
To conclude the testing process, it is important to produce a formal
record of what testing actions were taken, by whom, when they were
performed, and details of the test findings. It is wise to agree on an acceptable format for the report which is useful to all concerned parties,
which may include developers, project management, business owners, IT department, audit, and compliance.
The report should be clear to the business owner in identifying where
material risks exist and sufficient to get their backing for subsequent
mitigation actions. The report should also be clear to the developer in
pin-pointing the exact function that is affected by the vulnerability and
associated recommendations for resolving issues in a language that
the developer will understand. The report should also allow another
security tester to reproduce the results. Writing the report should not
be overly burdensome on the security tester themselves. Security
testers are not generally renowned for their creative writing skills and
agreeing on a complex report can lead to instances where test results
do not get properly documented. Using a security test report template
can save time and ensure that results are documented accurately and
consistently, and are in a format that is suitable for the audience.
13
Testing Guide Introduction
Testing Techniques Explained
This section presents a high-level overview of various testing
techniques that can be employed when building a testing program. It does not present specific methodologies for these techniques as this information is covered in Chapter 3. This section is
included to provide context for the framework presented in the
next chapter and to highlight the advantages and disadvantages
of some of the techniques that should be considered. In particular,
we will cover:
• Manual Inspections & Reviews
• Threat Modeling
• Code Review
• Penetration Testing
Manual Inspections & Reviews
Overview
Manual inspections are human reviews that typically test the security implications of people, policies, and processes. Manual inspections can also include inspection of technology decisions such
as architectural designs. They are usually conducted by analyzing
documentation or performing interviews with the designers or
system owners.
While the concept of manual inspections and human reviews is
simple, they can be among the most powerful and effective techniques available. By asking someone how something works and
why it was implemented in a specific way, the tester can quickly
determine if any security concerns are likely to be evident. Manual inspections and reviews are one of the few ways to test the
software development life-cycle process itself and to ensure that
there is an adequate policy or skill set in place.
As with many things in life, when conducting manual inspections
and reviews it is recommended that a trust-but-verify model is
adopted. Not everything that the tester is shown or told will be
accurate.
Manual reviews are particularly good for testing whether people
understand the security process, have been made aware of policy,
and have the appropriate skills to design or implement a secure
application.
Other activities, including manually reviewing the documentation,
secure coding policies, security requirements, and architectural
designs, should all be accomplished using manual inspections.
Advantages:
• Requires no supporting technology
• Can be applied to a variety of situations
• Flexible
• Promotes teamwork
• Early in the SDLC
Disadvantages:
• Can be time consuming
• Supporting material not always available
• Requires significant human thought and skill to be effective
Threat Modeling
Overview
Threat modeling has become a popular technique to help system
designers think about the security threats that their systems and
applications might face. Therefore, threat modeling can be seen as
risk assessment for applications. In fact, it enables the designer to
develop mitigation strategies for potential vulnerabilities and helps
them focus their inevitably limited resources and attention on the
parts of the system that most require it. It is recommended that
all applications have a threat model developed and documented.
Threat models should be created as early as possible in the SDLC,
and should be revisited as the application evolves and development progresses.
To develop a threat model, we recommend taking a simple approach that follows the NIST 800-30 [11] standard for risk assessment. This approach involves:
• Decomposing the application – use a process of manual
inspection to understand how the application works, its assets,
functionality, and connectivity.
• Defining and classifying the assets – classify the assets into
tangible and intangible assets and rank them according to
business importance.
• Exploring potential vulnerabilities - whether technical,
operational,or management.
• Exploring potential threats – develop a realistic view of potential
attack vectors from an attacker’s perspective, by using threat
scenarios or attack trees.
• Creating mitigation strategies – develop mitigating controls for
each of the threats deemed to be realistic.
The output from a threat model itself can vary but is typically a
collection of lists and diagrams. The OWASP Code Review Guide
outlines an Application Threat Modeling methodology that can be
used as a reference for the testing applications for potential security flaws in the design of the application. There is no right or
wrong way to develop threat models and perform information risk
assessments on applications. [12].
Advantages:
• Practical attacker’s view of the system
• Flexible
• Early in the SDLC
Disadvantages:
• Relatively new technique
• Good threat models don’t automatically mean good software
Source Code Review
Overview
Source code review is the process of manually checking the source
code of a web application for security issues. Many serious security vulnerabilities cannot be detected with any other form of analysis or testing. As the popular saying goes “if you want to know
what’s really going on, go straight to the source.” Almost all security experts agree that there is no substitute for actually looking
at the code. All the information for identifying security problems
is there in the code somewhere. Unlike testing third party closed
14
Testing Guide Introduction
software such as operating systems, when testing web applications (especially if they have been developed in-house) the source
code should be made available for testing purposes.
Many unintentional but significant security problems are also extremely difficult to discover with other forms of analysis or testing, such as penetration testing, making source code analysis the
technique of choice for technical testing. With the source code, a
tester can accurately determine what is happening (or is supposed
to be happening) and remove the guess work of black box testing.
Examples of issues that are particularly conducive to being found
through source code reviews include concurrency problems, flawed
business logic, access control problems, and cryptographic weaknesses as well as backdoors, Trojans, Easter eggs, time bombs,
logic bombs, and other forms of malicious code. These issues often manifest themselves as the most harmful vulnerabilities in
web sites. Source code analysis can also be extremely efficient to
find implementation issues such as places where input validation
was not performed or when fail open control procedures may be
present. But keep in mind that operational procedures need to be
reviewed as well, since the source code being deployed might not
be the same as the one being analyzed herein [13].
Advantages:
• Completeness and effectiveness
• Accuracy
• Fast (for competent reviewers)
Disadvantages:
• Requires highly skilled security developers
• Can miss issues in compiled libraries
• Cannot detect run-time errors easily
• The source code actually deployed might differ from the one
being analyzed
For more on code review, checkout the OWASP code review project.
Penetration Testing
Overview
Penetration testing has been a common technique used to test
network security for many years. It is also commonly known as
black box testing or ethical hacking. Penetration testing is essentially the “art” of testing a running application remotely to find
security vulnerabilities, without knowing the inner workings of
the application itself. Typically, the penetration test team would
have access to an application as if they were users. The tester acts
like an attacker and attempts to find and exploit vulnerabilities. In
many cases the tester will be given a valid account on the system.
While penetration testing has proven to be effective in network
security, the technique does not naturally translate to applications. When penetration testing is performed on networks and
operating systems, the majority of the work is involved in finding
and then exploiting known vulnerabilities in specific technologies.
As web applications are almost exclusively bespoke, penetration
testing in the web application arena is more akin to pure research.
Penetration testing tools have been developed that automate the
process, but with the nature of web applications their effective-
ness is usually poor.
Many people today use web application penetration testing as
their primary security testing technique. Whilst it certainly has its
place in a testing program, we do not believe it should be considered as the primary or only testing technique. Gary McGraw in [14]
summed up penetration testing well when he said, “If you fail a
penetration test you know you have a very bad problem indeed. If
you pass a penetration test you do not know that you don’t have
a very bad problem”. However, focused penetration testing (i.e.,
testing that attempts to exploit known vulnerabilities detected in
previous reviews) can be useful in detecting if some specific vulnerabilities are actually fixed in the source code deployed on the
web site.
Advantages:
• Can be fast (and therefore cheap)
• Requires a relatively lower skill-set than source code review
• Tests the code that is actually being exposed
Disadvantages:
• Too late in the SDLC
• Front impact testing only.
The Need for a Balanced Approach
With so many techniques and approaches to testing the security of
web applications it can be difficult to understand which techniques
to use and when to use them. Experience shows that there is no
right or wrong answer to the question of exactly what techniques
should be used to build a testing framework. In fact all techniques
should probably be used to test all the areas that need to be tested.
Although it is clear that there is no single technique that can be
performed to effectively cover all security testing and ensure that
all issues have been addressed, many companies adopt only one
approach. The approach used has historically been penetration
testing. Penetration testing, while useful, cannot effectively address many of the issues that need to be tested. It is simply “too
little too late” in the software development life cycle (SDLC).
The correct approach is a balanced approach that includes several
techniques, from manual reviews to technical testing. A balanced
approach should cover testing in all phases of the SDLC. This approach leverages the most appropriate techniques available depending on the current SDLC phase.
Of course there are times and circumstances where only one technique is possible. For example, a test on a web application that has
already been created, but where the testing party does not have
access to the source code. In this case, penetration testing is clearly
better than no testing at all. However, the testing parties should be
encouraged to challenge assumptions, such as no access to source
code, and to explore the possibility of more complete testing.
A balanced approach varies depending on many factors, such as
the maturity of the testing process and corporate culture. It is recommended that a balanced testing framework should look something like the representations shown in Figure 3 and Figure 4. The
following figure shows a typical proportional representation over-
15
Testing Guide Introduction
laid onto the software development life cycle. In keeping with research and experience, it is essential that companies place a higher
emphasis on the early stages of development.
10
Figure 3: Proportion of Test Effort in SDLC
5%
0-1
%1
5
-1
10 35
%
DEFINE
DESIGN
DEVELOP
MAINTAIN
12 -
15
-
35%
DEPLOY
‘Example 1: Magic Parameters’
Imagine a simple web application that accepts a name-value pair of
“magic” and then the value. For simplicity, the GET request may be:
http://www.host/application?magic=value
To further simplify the example, the values in this case can only be ASCII characters a – z (upper or lowercase) and integers 0 – 9.
The designers of this application created an administrative backdoor
during testing, but obfuscated it to prevent the casual observer from
discovering it. By submitting the value sf8g7sfjdsurtsdieerwqredsgnfg8d (30 characters), the user will then be logged in and presented
with an administrative screen with total control of the application. The
HTTP request is now:
http://www.host/application?magic= sf8g7sfjdsurtsdieerwqredsgnfg8d
%
25
Given that all of the other parameters were simple two- and
three-characters fields, it is not possible to start guessing combinations at approximately 28 characters. A web application scanner will
need to brute force (or guess) the entire key space of 30 characters.
That is up to 30^28 permutations, or trillions of HTTP requests. That
is an electron in a digital haystack.
The code for this exemplar Magic Parameter check may look like the
following:
The following figure shows a typical proportional representation
overlaid onto testing techniques.
Figure 4: Proportion of Test Effort According to Test Technique
PROCESS REVIEWS
& MANUAL INSPECTIONS
CODE REVIEW
SECURITY TESTING
A Note about Web Application Scanners
Many organizations have started to use automated web application
scanners. While they undoubtedly have a place in a testing program,
some fundamental issues need to be highlighted about why it is believed that automating black box testing is not (or will ever be) effective. However, highlighting these issues should not discourage the use
of web application scanners. Rather, the aim is to ensure the limitations are understood and testing frameworks are planned appropriately.
Important: OWASP is currently working to develop a web application
scanner bench marking platform. The following examples show why
automated black box testing is not effective.
public void doPost( HttpServletRequest request, HttpServletResponse response)
{
String magic = “sf8g7sfjdsurtsdieerwqredsgnfg8d”;
boolean admin = magic.equals( request.getParameter(“magic”));
if (admin) doAdmin( request, response);
else …. // normal processing
}
By looking in the code, the vulnerability practically leaps off the page
as a potential problem.
Example 2: Bad Cryptography
Cryptography is widely used in web applications. Imagine that a developer decided to write a simple cryptography algorithm to sign a user
in from site A to site B automatically. In his/her wisdom, the developer
decides that if a user is logged into site A, then he/she will generate
a key using an MD5 hash function that comprises: Hash { username :
date }
When a user is passed to site B, he/she will send the key on the query
string to site B in an HTTP re-direct. Site B independently computes
the hash, and compares it to the hash passed on the request. If they
match, site B signs the user in as the user they claim to be.
As the scheme is explained the inadequacies can be worked out. Anyone that figures out the scheme (or is told how it works, or downloads
the information from Bugtraq) can log in as any user. Manual inspection, such as a review or code inspection, would have uncovered this
security issue quickly. A black-box web application scanner would not
have uncovered the vulnerability. It would have seen a 128-bit hash
that changed with each user, and by the nature of hash functions, did
not change in any predictable way.
16
Testing Guide Introduction
A Note about Static Source Code Review Tools
Many organizations have started to use static source code scanners.
While they undoubtedly have a place in a comprehensive testing program, it is necessary to highlight some fundamental issues about why
this approach is not effective when used alone. Static source code
analysis alone cannot identify issues due to flaws in the design, since
it cannot understand the context in which the code is constructed.
Source code analysis tools are useful in determining security issues
due to coding errors, however significant manual effort is required to
validate the findings.
Deriving Security Test Requirements
To have a successful testing program, one must know what the testing objectives are. These objectives are specified by the security requirements. This section discusses in detail how to document requirements for security testing by deriving them from applicable standards
and regulations, and from positive and negative application requirements. It also discusses how security requirements effectively drive
security testing during the SDLC and how security test data can be
used to effectively manage software security risks.
Testing Objectives
One of the objectives of security testing is to validate that security
controls operate as expected. This is documented via security requirements that describe the functionality of the security control. At a
high level, this means proving confidentiality, integrity, and availability
of the data as well as the service. The other objective is to validate
that security controls are implemented with few or no vulnerabilities.
These are common vulnerabilities, such as the OWASP Top Ten, as
well as vulnerabilities that have been previously identified with security assessments during the SDLC, such as threat modelling, source
code analysis, and penetration test.
Security Requirements Documentation
The first step in the documentation of security requirements is to
understand the business requirements. A business requirement
document can provide initial high-level information on the expected
functionality of the application. For example, the main purpose of an
application may be to provide financial services to customers or to allow goods to be purchased from an on-line catalog. A security section
of the business requirements should highlight the need to protect the
customer data as well as to comply with applicable security documentation such as regulations, standards, and policies.
A general checklist of the applicable regulations, standards, and policies is a good preliminary security compliance analysis for web applications. For example, compliance regulations can be identified by
checking information about the business sector and the country or
state where the application will operate. Some of these compliance
guidelines and regulations might translate into specific technical requirements for security controls. For example, in the case of financial
applications, the compliance with FFIEC guidelines for authentication
[15] requires that financial institutions implement applications that
mitigate weak authentication risks with multi-layered security control and multi-factor authentication.
Applicable industry standards for security need also to be captured by
the general security requirement checklist. For example, in the case
of applications that handle customer credit card data, the compliance
with the PCI DSS [16] standard forbids the storage of PINs and CVV2
data and requires that the merchant protect magnetic strip data in
storage and transmission with encryption and on display by masking. Such PCI DSS security requirements could be validated via source
code analysis.
Another section of the checklist needs to enforce general requirements for compliance with the organization’s information security
standards and policies. From the functional requirements perspective, requirements for the security control need to map to a specific
section of the information security standards. An example of such requirement can be: “a password complexity of six alphanumeric characters must be enforced by the authentication controls used by the
application.” When security requirements map to compliance rules a
security test can validate the exposure of compliance risks. If violation
with information security standards and policies are found, these will
result in a risk that can be documented and that the business has to
manage. Since these security compliance requirements are enforceable, they need to be well documented and validated with security
tests.
Security Requirements Validation
From the functionality perspective, the validation of security requirements is the main objective of security testing. From the risk management perspective, the validation of security requirements is the
objective of information security assessments. At a high level, the
main goal of information security assessments is the identification of
gaps in security controls, such as lack of basic authentication, authorization, or encryption controls. More in depth, the security assessment objective is risk analysis, such as the identification of potential
weaknesses in security controls that ensure the confidentiality, integrity, and availability of the data. For example, when the application
deals with personal identifiable information (PII) and sensitive data,
the security requirement to be validated is the compliance with the
company information security policy requiring encryption of such
data in transit and in storage. Assuming encryption is used to protect
the data, encryption algorithms and key lengths need to comply with
the organization encryption standards. These might require that only
certain algorithms and key lengths could be used. For example, a security requirement that can be security tested is verifying that only
allowed ciphers are used (e.g., SHA-256, RSA, AES) with allowed minimum key lengths (e.g., more than 128 bit for symmetric and more
than 1024 for asymmetric encryption).
From the security assessment perspective, security requirements can
be validated at different phases of the SDLC by using different artifacts and testing methodologies. For example, threat modeling focuses on identifying security flaws during design, secure code analysis
and reviews focus on identifying security issues in source code during
development, and penetration testing focuses on identifying vulnerabilities in the application during testing or validation.
Security issues that are identified early in the SDLC can be documented in a test plan so they can be validated later with security tests. By
combining the results of different testing techniques, it is possible to
derive better security test cases and increase the level of assurance
of the security requirements. For example, distinguishing true vulnerabilities from the un-exploitable ones is possible when the results of
penetration tests and source code analysis are combined. Considering
the security test for a SQL injection vulnerability, for example, a black
box test might first involve a scan of the application to fingerprint the
vulnerability. The first evidence of a potential SQL injection vulnerability that can be validated is the generation of a SQL exception. A further
17
Testing Guide Introduction
validation of the SQL vulnerability might involve manually injecting
attack vectors to modify the grammar of the SQL query for an information disclosure exploit. This might involve a lot of trial-and-error
analysis until the malicious query is executed. Assuming the tester
has the source code, she might learn from the source code analysis
on how to construct the SQL attack vector that can exploit the vulnerability (e.g., execute a malicious query returning confidential data
to unauthorized user).
Threats and Countermeasures Taxonomies
A threat and countermeasure classification, which takes into consideration root causes of vulnerabilities, is the critical factor in verifying that security controls are designed, coded, and built to mitigate the impact of the exposure of such vulnerabilities. In the case
of web applications, the exposure of security controls to common
vulnerabilities, such as the OWASP Top Ten, can be a good starting
point to derive general security requirements. More specifically, the
web application security frame [17] provides a classification (e.g.
taxonomy) of vulnerabilities that can be documented in different
guidelines and standards and validated with security tests.
The focus of a threat and countermeasure categorization is to define
security requirements in terms of the threats and the root cause of
the vulnerability. A threat can be categorized by using STRIDE [18]
as Spoofing, Tampering, Repudiation, Information disclosure, Denial
of service, and Elevation of privilege. The root cause can be categorized as security flaw in design, a security bug in coding, or an issue
due to insecure configuration. For example, the root cause of weak
authentication vulnerability might be the lack of mutual authentication when data crosses a trust boundary between the client and
server tiers of the application. A security requirement that captures
the threat of non-repudiation during an architecture design review
allows for the documentation of the requirement for the countermeasure (e.g., mutual authentication) that can be validated later on
with security tests.
A threat and countermeasure categorization for vulnerabilities can
also be used to document security requirements for secure coding
such as secure coding standards. An example of a common coding
error in authentication controls consists of applying an hash function to encrypt a password, without applying a seed to the value.
From the secure coding perspective, this is a vulnerability that affects the encryption used for authentication with a vulnerability
root cause in a coding error. Since the root cause is insecure coding
the security requirement can be documented in secure coding standards and validated through secure code reviews during the development phase of the SDLC.
Security Testing and Risk Analysis
Security requirements need to take into consideration the severity
of the vulnerabilities to support a risk mitigation strategy. Assuming
that the organization maintains a repository of vulnerabilities found
in applications (i.e, a vulnerability knowledge base), the security
issues can be reported by type, issue, mitigation, root cause, and
mapped to the applications where they are found. Such a vulnerability knowledge base can also be used to establish a metrics to analyze the effectiveness of the security tests throughout the SDLC.
For example, consider an input validation issue, such as a SQL injection, which was identified via source code analysis and reported with a coding error root cause and input validation vulnerabil-
ity type. The exposure of such vulnerability can be assessed via a
penetration test, by probing input fields with several SQL injection
attack vectors. This test might validate that special characters are
filtered before hitting the database and mitigate the vulnerability.
By combining the results of source code analysis and penetration
testing it is possible to determine the likelihood and exposure of the
vulnerability and calculate the risk rating of the vulnerability. By reporting vulnerability risk ratings in the findings (e.g., test report) it is
possible to decide on the mitigation strategy. For example, high and
medium risk vulnerabilities can be prioritized for remediation, while
low risk can be fixed in further releases.
By considering the threat scenarios of exploiting common vulnerabilities it is possible to identify potential risks that the application
security control needs to be security tested for. For example, the
OWASP Top Ten vulnerabilities can be mapped to attacks such as
phishing, privacy violations, identify theft, system compromise,
data alteration or data destruction, financial loss, and reputation
loss. Such issues should be documented as part of the threat
scenarios. By thinking in terms of threats and vulnerabilities, it
is possible to devise a battery of tests that simulate such attack
scenarios. Ideally, the organization vulnerability knowledge base
can be used to derive security risk driven tests cases to validate
the most likely attack scenarios. For example, if identity theft is
considered high risk, negative test scenarios should validate the
mitigation of impacts deriving from the exploit of vulnerabilities
in authentication, cryptographic controls, input validation, and authorization controls.
Deriving Functional and Non Functional
Test Requirements
Functional Security Requirements
From the perspective of functional security requirements, the applicable standards, policies and regulations drive both the need for
a type of security control as well as the control functionality. These
requirements are also referred to as “positive requirements”, since
they state the expected functionality that can be validated through
security tests. Examples of positive requirements are: “the application will lockout the user after six failed log on attempts” or
“passwords need to be a minimum of six alphanumeric characters”.
The validation of positive requirements consists of asserting the
expected functionality and can be tested by re-creating the testing
conditions and running the test according to predefined inputs. The
results are then shown as as a fail or pass condition.
In order to validate security requirements with security tests, security requirements need to be function driven and they need to
highlight the expected functionality (the what) and implicitly the
implementation (the how). Examples of high-level security design
requirements for authentication can be:
• Protect user credentials and shared secrets in transit and in
storage
• Mask any confidential data in display (e.g., passwords, accounts)
• Lock the user account after a certain number of failed log in
attempts
• Do not show specific validation errors to the user as a result of a
failed log on
• Only allow passwords that are alphanumeric, include special
characters and six characters minimum length, to limit the attack
surface
18
Testing Guide Introduction
• Allow for password change functionality only to authenticated
users by validating the old password, the new password, and the
user answer to the challenge question, to prevent brute forcing of
a password via password change.
• The password reset form should validate the user’s username and
the user’s registered email before sending the temporary
password to the user via email. The temporary password issued
should be a one time password. A link to the password reset web
page will be sent to the user. The password reset web page should
validate the user temporary password, the new password, as well
as the user answer to the challenge question.
Risk Driven Security Requirements
Security tests need also to be risk driven, that is they need to validate the application for unexpected behavior. These are also called
“negative requirements”, since they specify what the application
should not do.
Examples of negative requirements are:
• The application should not allow for the data to be altered or
destroyed
• The application should not be compromised or misused for
unauthorized financial transactions by a malicious user.
Negative requirements are more difficult to test, because there is
no expected behavior to look for. This might require a threat analyst to come up with unforeseeable input conditions, causes, and
effects. This is where security testing needs to be driven by risk
analysis and threat modeling. The key is to document the threat
scenarios and the functionality of the countermeasure as a factor
to mitigate a threat.
For example, in the case of authentication controls, the following
security requirements can be documented from the threats and
countermeasure perspective:
• Encrypt authentication data in storage and transit to mitigate risk
of information disclosure and authentication protocol attacks
• Encrypt passwords using non reversible encryption such as using
a digest (e.g., HASH) and a seed to prevent dictionary attacks
• Lock out accounts after reaching a log on failure threshold and
enforce password complexity to mitigate risk of brute force
password attacks
• Display generic error messages upon validation of credentials to
mitigate risk of account harvesting or enumeration
• Mutually authenticate client and server to prevent non-repudiation
and Man In the Middle (MiTM) attacks
Threat modeling tools such as threat trees and attack libraries can
be useful to derive the negative test scenarios. A threat tree will
assume a root attack (e.g., attacker might be able to read other users’ messages) and identify different exploits of security controls
(e.g., data validation fails because of a SQL injection vulnerability)
and necessary countermeasures (e.g., implement data validation
and parametrized queries) that could be validated to be effective
in mitigating such attacks.
Deriving Security Test Requirements Through Use and Misuse
Cases
A prerequisite to describing the application functionality is to un-
derstand what the application is supposed to do and how. This can
be done by describing use cases. Use cases, in the graphical form
as commonly used in software engineering, show the interactions
of actors and their relations. They help to identify the actors in the
application, their relationships, the intended sequence of actions
for each scenario, alternative actions, special requirements, preconditions and and post-conditions.
Similar to use cases, misuse and abuse cases [19] describe unintended and malicious use scenarios of the application. These misuse cases provide a way to describe scenarios of how an attacker
could misuse and abuse the application. By going through the individual steps in a use scenario and thinking about how it can be
maliciously exploited, potential flaws or aspects of the application
that are not well-defined can be discovered. The key is to describe
all possible or, at least, the most critical use and misuse scenarios.
Misuse scenarios allow the analysis of the application from the attacker’s point of view and contribute to identifying potential vulnerabilities and the countermeasures that need to be implemented to
mitigate the impact caused by the potential exposure to such vulnerabilities. Given all of the use and abuse cases, it is important to
analyze them to determine which of them are the most critical ones
and need to be documented in security requirements. The identification of the most critical misuse and abuse cases drives the documentation of security requirements and the necessary controls
where security risks should be mitigated.
To derive security requirements from use and misuse case [20] it is
important to define the functional scenarios and the negative scenarios and put these in graphical form. In the case of derivation of
security requirements for authentication, for example, the following
step-by-step methodology can be followed.
Step 1: Describe the Functional Scenario: User authenticates by
supplying a username and password. The application grants access
to users based upon authentication of user credentials by the application and provides specific errors to the user when validation fails.
Step 2: Describe the Negative Scenario: Attacker breaks the authentication through a brute force or dictionary attack of passwords and account harvesting vulnerabilities in the application.
The validation errors provide specific information to an attacker to
guess which accounts are actually valid registered accounts (usernames). Then the attacker will try to brute force the password for
such a valid account. A brute force attack to four minimum length
all digit passwords can succeed with a limited number of attempts
(i.e., 10^4).
Step 3: Describe Functional and Negative Scenarios With Use and
Misuse Case: The graphical example in Figure below depicts the
derivation of security requirements via use and misuse cases. The
functional scenario consists of the user actions (enteringa username and password) and the application actions (authenticating
the user and providing an error message if validation fails). The misuse case consists of the attacker actions, i.e. trying to break authentication by brute forcing the password via a dictionary attack and by
guessing the valid usernames from error messages. By graphically
representing the threats to the user actions (misuses), it is possible
to derive the countermeasures as the application actions that mitigate such threats.
19
Testing Guide Introduction
the application build, the results of the static and dynamic analysis
should be reviewed and validated.
USER
Enter
username
and
password
Includes
User
authentiction
Brute force
authentication
Includes
APPLICATION /
SERVER
Show
generic
error
message
Harvest
(guess)
valid user
accounts
HACKER /
MALICIOUS
USER
Includes
Look account
after N
failed login
attempts
Dictionary
attacks
Includes
Validate
password
minimum lenght
and complexity
Step 4: Elicit The Security Requirements. In this case, the following
security requirements for authentication are derived:
1) Passwords need to be alphanumeric, lower and upper case and
minimum of seven character length
2) Accounts need to lockout after five unsuccessful log in attempt
3) Log in error messages need to be generic
These security requirements need to be documented and tested.
Security Tests Integrated in Development and
Testing Workflows
Security Testing in the Development Workflow
Security testing during the development phase of the SDLC represents the first opportunity for developers to ensure that the individual software components they have developed are security
tested before they are integrated with other components and built
into the application. Software components might consist of software artifacts such as functions, methods, and classes, as well
as application programming interfaces, libraries, and executable
files. For security testing, developers can rely on the results of the
source code analysis to verify statically that the developed source
code does not include potential vulnerabilities and is compliant with
the secure coding standards. Security unit tests can further verify
dynamically (i.e., at run time) that the components function as expected. Before integrating both new and existing code changes in
The validation of source code before integration in application builds
is usually the responsibility of the senior developer. Such senior developers are also the subject matter experts in software security
and their role is to lead the secure code review. They must make decisions on whether to accept the code to be released in the application build or to require further changes and testing. This secure code
review workflow can be enforced via formal acceptance as well as a
check in a workflow management tool. For example, assuming the
typical defect management workflow used for functional bugs, security bugs that have been fixed by a developer can be reported on a
defect or change management system. The build master can look at
the test results reported by the developers in the tool and grant approvals for checking in the code changes into the application build.
Security Testing in the Test Workflow
After components and code changes are tested by developers and
checked in to the application build, the most likely next step in the
software development process workflow is to perform tests on the
application as a whole entity. This level of testing is usually referred
to as integrated test and system level test. When security tests are
part of these testing activities they can be used to validate both the
security functionality of the application as a whole, as well as the
exposure to application level vulnerabilities.These security tests on
the application include both white box testing, such as source code
analysis, and black box testing, such as penetration testing. Gray
box testing is similar to Black box testing. In a gray box testing it
is assumed that the tester has some partial knowledge about the
session management of the application, and that should help in understanding whether the log out and timeout functions are properly
secured.
The target for the security tests is the complete system that will be
potentially attacked and includes both the whole source code and
the executable. One peculiarity of security testing during this phase
is that it is possible for security testers to determine whether vulnerabilities can be exploited and expose the application to real risks.
These include common web application vulnerabilities, as well as
security issues that have been identified earlier in the SDLC with
other activities such as threat modeling, source code analysis, and
secure code reviews.
Usually testing engineers, rather then software developers, perform security tests when the application is in scope for integration
system tests. Such testing engineers have security knowledge of
web application vulnerabilities, black box and white box security
testing techniques, and own the validation of security requirements
in this phase. In order to perform such security tests, it is a prerequisite that security test cases are documented in the security testing
guidelines and procedures.
A testing engineer who validates the security of the application in
the integrated system environment might release the application
for testing in the operational environment (e.g., user acceptance
tests). At this stage of the SDLC (i.e., validation), the application
functional testing is usually a responsibility of QA testers, while
white-hat hackers or security consultants are usually responsible
for security testing. Some organizations rely on their own specialized ethical hacking team to conduct such tests when a third party
20
Testing Guide Introduction
assessment is not required (such as for auditing purposes).
Since these tests are the last resort for fixing vulnerabilities before the application is released to production, it is important that
such issues are addressed as recommended by the testing team.
The recommendations can include code, design, or configuration
change. At this level, security auditors and information security officers discuss the reported security issues and analyze the potential
risks according to information risk management procedures. Such
procedures might require the development team to fix all high risk
vulnerabilities before the application can be deployed, unless such
risks are acknowledged and accepted.
Developers’ Security Tests
Security Testing in the Coding Phase: Unit Tests
From the developer’s perspective, the main objective of security
tests is to validate that code is being developed in compliance with
secure coding standards requirements. Developers’ own coding
artifacts (such as functions, methods, classes, APIs, and libraries)
need to be functionally validated before being integrated into the
application build.
The security requirements that developers have to follow should be
documented in secure coding standards and validated with static
and dynamic analysis. If the unit test activity follows a secure code
review, unit tests can validate that code changes required by secure code reviews are properly implemented. Secure code reviews
and source code analysis through source code analysis tools help
developers in identifying security issues in source code as it is developed. By using unit tests and dynamic analysis (e.g., debugging)
developers can validate the security functionality of components as
well as verify that the countermeasures being developed mitigate
any security risks previously identified through threat modeling and
source code analysis.
the case of security functional tests, unit level tests can test the
functionality of security controls at the software component level, such as functions, methods, or classes. For example, a test case
could validate input and output validation (e.g., variable sanitation)
and boundary checks for variables by asserting the expected functionality of the component.
The threat scenarios identified with use and misuse cases can be
used to document the procedures for testing software components. In the case of authentication components, for example, security unit tests can assert the functionality of setting an account
lockout as well as the fact that user input parameters cannot be
abused to bypass the account lockout (e.g., by setting the account
lockout counter to a negative number).
At the component level, security unit tests can validate positive assertions as well as negative assertions, such as errors and exception handling. Exceptions should be caught without leaving the system in an insecure state, such as potential denial of service caused
by resources not being de-allocated (e.g., connection handles not
closed within a final statement block), as well as potential elevation
of privileges (e.g., higher privileges acquired before the exception is
thrown and not re-set to the previous level before exiting the function). Secure error handling can validate potential information disclosure via informative error messages and stack traces.
Unit level security test cases can be developed by a security engineer who is the subject matter expert in software security and is
also responsible for validating that the security issues in the source
code have been fixed and can be checked into the integrated system
build. Typically, the manager of the application builds also makes
sure that third-party libraries and executable files are security assessed for potential vulnerabilities before being integrated in the
application build.
A good practice for developers is to build security test cases as a
generic security test suite that is part of the existing unit testing
framework. A generic security test suite could be derived from previously defined use and misuse cases to security test functions,
methods and classes. A generic security test suite might include
security test cases to validate both positive and negative requirements for security controls such as:
Threat scenarios for common vulnerabilities that have root causes
in insecure coding can also be documented in the developer’s security testing guide. When a fix is implemented for a coding defect
identified with source code analysis, for example, security test cases can verify that the implementation of the code change follows
the secure coding requirements documented in the secure coding
standards.
• Identity, Authentication & Access Control
• Input Validation & Encoding
• Encryption
• User and Session Management
• Error and Exception Handling
• Auditing and Logging
Source code analysis and unit tests can validate that the code
change mitigates the vulnerability exposed by the previously identified coding defect. The results of automated secure code analysis
can also be used as automatic check-in gates for version control, for
example software artifacts cannot be checked into the build with
high or medium severity coding issues.
Developers empowered with a source code analysis tool integrated
into their IDE, secure coding standards, and a security unit testing
framework can assess and verify the security of the software components being developed. Security test cases can be run to identify
potential security issues that have root causes in source code: besides input and output validation of parameters entering and exiting
the components, these issues include authentication and authorization checks done by the component, protection of the data within
the component, secure exception and error handling, and secure
auditing and logging. Unit test frameworks such as Junit, Nunit,
and CUnit can be adapted to verify security test requirements. In
Functional Testers’ Security Tests
Security Testing During the Integration and Validation Phase:
Integrated System Tests and Operation Tests
The main objective of integrated system tests is to validate the “defense in depth” concept, that is, that the implementation of security controls provides security at different layers. For example, the
lack of input validation when calling a component integrated with
the application is often a factor that can be tested with integration
testing.
The integration system test environment is also the first environ-
21
Testing Guide Introduction
ment where testers can simulate real attack scenarios as can be
potentially executed by a malicious external or internal user of the
application. Security testing at this level can validate whether vulnerabilities are real and can be exploited by attackers. For example,
a potential vulnerability found in source code can be rated as high
risk because of the exposure to potential malicious users, as well
as because of the potential impact (e.g., access to confidential information).
with the exception of the data (e.g., test data is used in place of real
data). A characteristic of security testing in UAT is testing for security configuration issues. In some cases these vulnerabilities might
represent high risks. For example, the server that hosts the web
application might not be configured with minimum privileges, valid
SSL certificate and secure configuration, essential services disabled
and web root directory not cleaned from test and administration
web pages.
Real attack scenarios can be tested with both manual testing techniques and penetration testing tools. Security tests of this type are
also referred to as ethical hacking tests. From the security testing
perspective, these are risk driven tests and have the objective of
testing the application in the operational environment. The target
is the application build that is representative of the version of the
application being deployed into production.
Security Test Data Analysis and Reporting
Including security testing in the integration and validation phase
is critical to identifying vulnerabilities due to integration of components as well as validating the exposure of such vulnerabilities. Application security testing requires a specialized set of
skills, including both software and security knowledge, that are
not typical of security engineers.As a result organizations are often required to security-train their software developers on ethical
hacking techniques, security assessment procedures and tools.
A realistic scenario is to develop such resources in-house and
document them in security testing guides and procedures that
take into account the developer’s security testing knowledge.
A so called “security test cases cheat list or check-list”, for example,
can provide simple test cases and attack vectors that can be used
by testers to validate exposure to common vulnerabilities such as
spoofing, information disclosures, buffer overflows, format strings,
SQL injection and XSS injection, XML, SOAP, canonicalization issues,
denial of service and managed code and ActiveX controls (e.g., .NET).
A first battery of these tests can be performed manually with a very
basic knowledge of software security.
The first objective of security tests might be the validation of a set
of minimum security requirements. These security test cases might
consist of manually forcing the application into error and exceptional states and gathering knowledge from the application behavior.
For example, SQL injection vulnerabilities can be tested manually by
injecting attack vectors through user input and by checking if SQL
exceptions are thrown back the user. The evidence of a SQL exception error might be a manifestation of a vulnerability that can be
exploited.
A more in-depth security test might require the tester’s knowledge of specialized testing techniques and tools. Besides source
code analysis and penetration testing, these techniques include, for
example, source code and binary fault injection, fault propagation
analysis and code coverage, fuzz testing, and reverse engineering.
The security testing guide should provide procedures and recommend tools that can be used by security testers to perform such
in-depth security assessments.
The next level of security testing after integration system tests is to
perform security tests in the user acceptance environment. There
are unique advantages to performing security tests in the operational environment. The user acceptance tests environment (UAT)
is the one that is most representative of the release configuration,
Goals for Security Test Metrics and Measurements
Defining the goals for the security testing metrics and measurements is a prerequisite for using security testing data for risk analysis and management processes. For example, a measurement
such as the total number of vulnerabilities found with security tests
might quantify the security posture of the application. These measurements also help to identify security objectives for software security testing.For example, reducing the number of vulnerabilities to
an acceptable number (minimum) before the application is deployed
into production.
Another manageable goal could be to compare the application
security posture against a baseline to assess improvements in
application security processes. For example, the security metrics
baseline might consist of an application that was tested only with
penetration tests. The security data obtained from an application
that was also security tested during coding should show an improvement (e.g., fewer number of vulnerabilities) when compared
with the baseline.
In traditional software testing, the number of software defects,
such as the bugs found in an application, could provide a measure of
software quality. Similarly, security testing can provide a measure
of software security. From the defect management and reporting
perspective, software quality and security testing can use similar
categorizations for root causes and defect remediation efforts.
From the root cause perspective, a security defect can be due to an
error in design (e.g., security flaws) or due to an error in coding (e.g.,
security bug). From the perspective of the effort required to fix a
defect, both security and quality defects can be measured in terms
of developer hours to implement the fix, the tools and resources
required to fix, and the cost to implement the fix.
A characteristic of security test data, compared to quality data,
is the categorization in terms of the threat, the exposure of
the vulnerability, and the potential impact posed by the vulnerability to determine the risk. Testing applications for security consists of managing technical risks to make sure that
the application countermeasures meet acceptable levels.
For this reason, security testing data needs to support the security risk strategy at critical checkpoints during the SDLC.
For example, vulnerabilities found in source code with source code
analysis represent an initial measure of risk. A measure of risk
(e.g., high, medium, low) for the vulnerability can be calculated by
determining the exposure and likelihood factors and by validating
the vulnerability with penetration tests. The risk metrics associated to vulnerabilities found with security tests empower business
management to make risk management decisions, such as to decide whether risks can be accepted, mitigated, or transferred at
different levels within the organization (e.g., business as well as
technical risks).
22
Testing Guide Introduction
When evaluating the security posture of an application it is important to take into consideration certain factors, such as the
size of the application being developed. Application size has
been statistically proven to be related to the number of issues
found in the application during testing. One measure of application size is the number of lines of code (LOC) of the application.
Typically, software quality defects range from about 7 to 10 defects
per thousand lines of new and changed code [21]. Since testing
can reduce the overall number by about 25% with one test alone,
it is logical for larger size applications to be tested more often than
smaller size applications.
When security testing is done in several phases of the SDLC, the
test data can prove the capability of the security tests in detecting vulnerabilities as soon as they are introduced. The test data can
also prove the effectiveness of removing the vulnerabilities by implementing countermeasures at different checkpoints of the SDLC.
A measurement of this type is also defined as “containment metrics” and provides a measure of the ability of a security assessment performed at each phase of the development process to maintain security within each phase.
These containment metrics are also a critical factor in lowering the
cost of fixing the vulnerabilities. It is less expensive to deal with
vulnerabilities in the same phase of the SDLC that they are found,
rather then fixing them later in another phase.
Security test metrics can support security risk, cost, and defect
management analysis when they are associated with tangible and
timed goals such as:
• Reducing the overall number of vulnerabilities by 30%
• Fixing security issues by a certain deadline (e.g., before beta
release)
Security test data can be absolute, such as the number of vulnerabilities detected during manual code review, as well as comparative,
such as the number of vulnerabilities detected in code reviews compared to penetration tests. To answer questions about the quality
of the security process, it is important to determine a baseline for
what could be considered acceptable and good. Security test data
can also support specific objectives of the security analysis. These
objects could be compliance with security regulations and information security standards, management of security processes, the
identification of security root causes and process improvements,
and security cost benefit analysis.
When security test data is reported it has to provide metrics to support the analysis. The scope of the analysis is the interpretation of
test data to find clues about the security of the software being produced as well the effectiveness of the process.
Some examples of clues supported by security test data can be:
• Are vulnerabilities reduced to an acceptable level for release?
• How does the security quality of this product compare with
similar software products?
• Are all security test requirements being met?
• What are the major root causes of security issues?
• How numerous are security flaws compared to security bugs?
• Which security activity is most effective in finding vulnerabilities?
• Which team is more productive in fixing security defects
and vulnerabilities?
• Which percentage of overall vulnerabilities are high risk?
• Which tools are most effective in detecting security vulnerabilities?
• Which kind of security tests are most effective in finding
vulnerabilities (e.g., white box vs. black box) tests?
• How many security issues are found during secure code reviews?
• How many security issues are found during secure design
reviews?
In order to make a sound judgment using the testing data, it is important to have a good understanding of the testing process as well
as the testing tools. A tool taxonomy should be adopted to decide
which security tools to use. Security tools can be qualified as being
good at finding common known vulnerabilities targeting different
artifacts.
The issue is that the unknown security issues are not tested. The fact
that a security test is clear of issues does not mean that the software
or application is good. Some studies [22] have demonstrated that, at
best, tools can only find 45% of overall vulnerabilities.
Even the most sophisticated automation tools are not a match for
an experienced security tester. Just relying on successful test results from automation tools will give security practitioners a false
sense of security.Typically, the more experienced the security testers are with the security testing methodology and testing tools,
the better the results of the security test and analysis will be. It is
important that managers making an investment in security testing
tools also consider an investment in hiring skilled human resources
as well as security test training.
Reporting Requirements
The security posture of an application can be characterized from the
perspective of the effect, such as number of vulnerabilities and the
risk rating of the vulnerabilities, as well as from the perspective of
the cause or origin, such as coding errors, architectural flaws, and
configuration issues.
Vulnerabilities can be classified according to different criteria.
The most commonly used vulnerability severity metric is the Forum
of Incident Response and Security Teams (FIRST) Common Vulnerability Scoring System (CVSS), which is currently in release version 2
with version 3 due for release shortly.
When reporting security test data the best practice is to include the
following information:
• The categorization of each vulnerability by type
• The security threat that the issue is exposed to
• The root cause of security issues (e.g., security bugs, security flaw)
• The testing technique used to find the issue
• The remediation of the vulnerability (e.g., the countermeasure)
• The severity rating of the vulnerability (High, Medium, Low and/
or CVSS score)
By describing what the security threat is, it will be possible to understand if and why the mitigation control is ineffective in mitigating the threat.
Reporting the root cause of the issue can help pinpoint what
needs to be fixed. In the case of a white box testing, for example,
the software security root cause of the vulnerability will be the
23
Testing Guide Introduction
offending source code.
References
Once issues are reported, it is also important to provide guidance to
the software developer on how to re-test and find the vulnerability.
This might involve using a white box testing technique (e.g., security
code review with a static code analyzer) to find if the code is vulnerable. If a vulnerability can be found via a black box technique (penetration test), the test report also needs to provide information on how to
validate the exposure of the vulnerability to the front end (e.g., client).
[1] T. DeMarco, Controlling Software Projects: Management,
Measurement and Estimation, Yourdon Press, 1982
[2] S. Payne, A Guide to Security Metrics - http://www.sans.org/
reading_room/whitepapers/auditing/55.php
[3] NIST, The economic impacts of inadequate infrastructure for
software testing - http://www.nist.gov/director/planning/upload/
report02-3.pdf
[4] Ross Anderson, Economics and Security Resource Page http://www.cl.cam.ac.uk/~rja14/econsec.html
[5] Denis Verdon, Teaching Developers To Fish - OWASP AppSec
NYC 2004
[6] Bruce Schneier, Cryptogram Issue #9 - https://www.schneier.
com/crypto-gram-0009.html
[7 Symantec, Threat Reports - http://www.symantec.com/
security_response/publications/threatreport.jsp
[8] FTC, The Gramm-Leach Bliley Act - http://business.ftc.gov/
privacy-and-security/gramm-leach-bliley-act
[9] Senator Peace and Assembly Member Simitian, SB 1386http://www.leginfo.ca.gov/pub/01-02/bill/sen/sb_1351-1400/
sb_1386_bill_20020926_chaptered.html
[10] European Union, Directive 96/46/EC on the protection of
individuals with regard to the processing of personal data and
on the free movement of such data - http://ec.europa.eu/justice/
policies/privacy/docs/95-46-ce/dir1995-46_part1_en.pdf
[11] NIST, Risk management guide for information technology
systems - http://csrc.nist.gov/publications/nistpubs/800-30-rev1/
sp800_30_r1.pdf
[12] SEI, Carnegie Mellon, Operationally Critical Threat, Asset,
and Vulnerability Evaluation (OCTAVE) - http://www.cert.org/
octave/
[13] Ken Thompson, Reflections on Trusting Trust, Reprinted
from Communication of the ACM - http://cm.bell-labs.com/who/
ken/trust.html
[14] Gary McGraw, Beyond the Badness-ometer - http://www.
drdobbs.com/security/beyond-the-badness-ometer/189500001
[15] FFIEC, Authentication in an Internet Banking Environment http://www.ffiec.gov/pdf/authentication_guidance.pdf
[16] PCI Security Standards Council, PCI Data Security Standard
- https://www.pcisecuritystandards.org/security_standards/index.
php
[17] MSDN, Cheat Sheet: Web Application Security Frame h t t p : //m s d n . m i c ro s o f t . c o m /e n - u s / l i b r a r y/m s 9 7 8 5 1 8 .
aspx#tmwacheatsheet_webappsecurityframe
[18] MSDN, Improving Web Application Security, Chapter 2,
Threat And Countermeasures - http://msdn.microsoft.com/en-us/
library/aa302418.aspx
[19] Sindre,G. Opdmal A., Capturing Security Requirements
Through Misuse Cases ‘ - http://folk.uio.no/nik/2001/21-sindre.
pdf
[20] Improving Security Across the Software Development
Lifecycle Task Force, Referred Data from Caper Johns, Software
Assessments, Benchmarks and Best Practices - http://www.
criminal-justice-careers.com/resources/SDLCFULL.pdf
[21] MITRE, Being Explicit About Weaknesses, Slide 30,
Coverage of CWE - http://cwe.mitre.org/documents/being-explicit/
BlackHatDC_BeingExplicit_Slides.ppt
[22] Marco Morana, Building Security Into The Software Life
Cycle, A Business Case - http://www.blackhat.com/presentations/
bh-usa-06/bh-us-06-Morana-R3.0.pdf
The information about how to fix the vulnerability should be detailed enough for a developer to implement a fix. It should provide
secure coding examples, configuration changes, and provide adequate references.
Finally, the severity rating contributes to the calculation of risk rating and helps to prioritize the remediation effort. Typically, assigning
a risk rating to the vulnerability involves external risk analysis based
upon factors such as impact and exposure.
Business Cases
For the security test metrics to be useful, they need to provide value back to the organization’s security test data stakeholders. The
stakeholders can include project managers, developers, information
security offices, auditors, and chief information officers. The value
can be in terms of the business case that each project stakeholder
has in terms of role and responsibility.
Software developers look at security test data to show that software
is coded more securely and efficiently. This allows them to make the
case for using source code analysis tools as well as following secure
coding standards and attending software security training.
Project managers look for data that allows them to successfully
manage and utilize security testing activities and resources according to the project plan. To project managers, security test data can
show that projects are on schedule and moving on target for delivery dates and are getting better during tests.
Security test data also helps the business case for security testing
if the initiative comes from information security officers (ISOs). For
example, it can provide evidence that security testing during the SDLC
does not impact the project delivery, but rather reduces the overall
workload needed to address vulnerabilities later in production.
To compliance auditors, security test metrics provide a level of
software security assurance and confidence that security standard
compliance is addressed through the security review processes
within the organization.
Finally, Chief Information Officers (CIOs) and Chief Information Security Officers (CISOs), who are responsible for the budget that needs to
be allocated in security resources, look for derivation of a cost benefit
analysis from security test data.This allows them to make informed
decisions on which security activities and tools to invest. One of the
metrics that supports such analysis is the Return On Investment
(ROI) in Security [23]. To derive such metrics from security test data,
it is important to quantify the differential between the risk due to the
exposure of vulnerabilities and the effectiveness of the security tests
in mitigating the security risk, and factor this gap with the cost of the
security testing activity or the testing tools adopted.
24
3
The OWASP Testing Framework
The OWASP Testing Framework
This section describes a typical testing framework that can be
developed within an organization. It can be seen as a reference
framework that comprises techniques and tasks that are
appropriate at various phases of the software development life
cycle (SDLC).
Overview
This section describes a typical testing framework that can be developed within an organization. It can be seen as a reference framework that comprises techniques and tasks that are appropriate at
various phases of the software development life cycle (SDLC). Companies and project teams can use this model to develop their own
testing framework and to scope testing services from vendors. This
framework should not be seen as prescriptive, but as a flexible approach that can be extended and molded to fit an organization’s
development process and culture.
This section aims to help organizations build a complete strategic
testing process, and is not aimed at consultants or contractors who
tend to be engaged in more tactical, specific areas of testing.
It is critical to understand why building an end-to-end testing
framework is crucial to assessing and improving software security.
In Writing Secure Code Howard and LeBlanc note that issuing a security bulletin costs Microsoft at least $100,000, and it costs their
customers collectively far more than that to implement the security
patches. They also note that the US government’s CyberCrime web
site (http://www.justice.gov/criminal/cybercrime/) details recent
criminal cases and the loss to organizations. Typical losses far exceed USD $100,000.
With economics like this, it is little wonder why software vendors
move from solely performing black box security testing, which can
only be performed on applications that have already been developed, to concentrate on testing in the early cycles of application
development such as definition, design, and development.
Many security practitioners still see security testing in the realm of
penetration testing. As discussed before, while penetration testing
has a role to play, it is generally inefficient at finding bugs and relies
excessively on the skill of the tester. It should only be considered as
an implementation technique, or to raise awareness of production
issues. To improve the security of applications, the security quality
of the software must be improved. That means testing the security
at the definition, design, develop, deploy, and maintenance stages,
and not relying on the costly strategy of waiting until code is completely built.
As discussed in the introduction of this document, there are many
development methodologies such as the Rational Unified Process,
eXtreme and Agile development, and traditional waterfall methodologies. The intent of this guide is to suggest neither a particular development methodology nor provide specific guidance that adheres
to any particular methodology. Instead, we are presenting a generic
development model, and the reader should follow it according to
their company process.
This testing framework consists of the following activities that
should take place:
• Before development begins
• During definition and design
• During development
• During deployment
• Maintenance and operations
Phase 1: Before Development Begins
Phase 1.1: Define a SDLC
Before application development starts an adequate SDLC must be
defined where security is inherent at each stage.
Phase 1.2: Review Policies and Standards
Ensure that there are appropriate policies, standards, and documentation in place. Documentation is extremely important as it gives development teams guidelines and policies that they can follow.
People can only do the right thing if they know what the right thing is.
If the application is to be developed in Java, it is essential that there
is a Java secure coding standard. If the application is to use cryptography, it is essential that there is a cryptography standard. No policies or standards can cover every situation that the development
team will face. By documenting the common and predictable issues,
there will be fewer decisions that need to be made during the development process.
Phase 1.3: Develop Measurement and Metrics Criteria and Ensure
Traceability
Before development begins, plan the measurement program. By
defining criteria that need to be measured, it provides visibility into
defects in both the process and product. It is essential to define
the metrics before development begins, as there may be a need to
modify the process in order to capture the data.
Phase 2: During Definition and Design
Phase 2.1: Review Security Requirements
Security requirements define how an application works from a security perspective. It is essential that the security requirements are
tested. Testing in this case means testing the assumptions that are
made in the requirements and testing to see if there are gaps in the
requirements definitions.
For example, if there is a security requirement that states that users
must be registered before they can get access to the whitepapers
25
The OWASP Testing Framework
section of a website, does this mean that the user must be registered with the system or should the user be authenticated? Ensure
that requirements are as unambiguous as possible.
When looking for requirements gaps, consider looking at security
mechanisms such as:
• User Management
• Authentication
• Authorization
• Data Confidentiality
• Integrity
• Accountability
• Session Management
• Transport Security
• Tiered System Segregation
• Legislative and standards compliance (including Privacy,
Government and Industry standards)
Phase 2.2: Review Design and Architecture
Applications should have a documented design and architecture.
This documentation can include models, textual documents, and
other similar artifacts. It is essential to test these artifacts to ensure
that the design and architecture enforce the appropriate level of security as defined in the requirements.
Identifying security flaws in the design phase is not only one of the
most cost-efficient places to identify flaws, but can be one of the
most effective places to make changes. For example, if it is identified that the design calls for authorization decisions to be made in
multiple places, it may be appropriate to consider a central authorization component. If the application is performing data validation
at multiple places, it may be appropriate to develop a central validation framework (ie, fixing input validation in one place, rather than in
hundreds of places, is far cheaper).
If weaknesses are discovered, they should be given to the system
architect for alternative approaches.
development. These are often smaller decisions that were either too
detailed to be described in the design, or issues where no policy or
standard guidance was offered. If the design and architecture were
not adequate, the developer will be faced with many decisions. If
there were insufficient policies and standards, the developer will be
faced with even more decisions.
Phase 3.1: Code Walk Through
The security team should perform a code walk through with the
developers, and in some cases, the system architects. A code walk
through is a high-level walk through of the code where the developers can explain the logic and flow of the implemented code. It allows the code review team to obtain a general understanding of the
code, and allows the developers to explain why certain things were
developed the way they were.
The purpose is not to perform a code review, but to understand at
a high level the flow, the layout, and the structure of the code that
makes up the application.
Phase 3.2: Code Reviews
Armed with a good understanding of how the code is structured
and why certain things were coded the way they were, the tester
can now examine the actual code for security defects.
Static code reviews validate the code against a set of checklists,
icluding:
• Business requirements for availability, confidentiality, and
integrity.
• OWASP Guide or Top 10 Checklists for technical exposures
(depending on the depth of the review).
• Specific issues relating to the language or framework in use, such
as the Scarlet paper for PHP or Microsoft Secure Coding checklists
for ASP.NET.
• Any industry specific requirements, such as Sarbanes-Oxley 404,
COPPA, ISO/IEC 27002, APRA, HIPAA, Visa Merchant guidelines,
or other regulatory regimes.
Phase 2.3: Create and Review UML Models
Once the design and architecture is complete, build Unified
Modeling Language (UML) models that describe how the application works. In some cases, these may already be available.
Use these models to confirm with the systems designers an exact
understanding of how the application works. If weaknesses are discovered, they should be given to the system architect for alternative
approaches.
In terms of return on resources invested (mostly time), static code
reviews produce far higher quality returns than any other security
review method and rely least on the skill of the reviewer. However,
they are not a silver bullet and need to be considered carefully within a full-spectrum testing regime.
Phase 2.4: Create and Review Threat Models
Armed with design and architecture reviews and the UML models
explaining exactly how the system works, undertake a threat modeling exercise. Develop realistic threat scenarios. Analyze the design
and architecture to ensure that these threats have been mitigated,
accepted by the business, or assigned to a third party, such as an
insurance firm. When identified threats have no mitigation strategies, revisit the design and architecture with the systems architect
to modify the design.
Phase 4: During Deployment
Phase 3: During Development
Theoretically, development is the implementation of a design. However, in the real world, many design decisions are made during code
For more details on OWASP checklists, please refer to OWASP Guide
for Secure Web Applications, or the latest edition of the OWASP Top 10.
Phase 4.1: Application Penetration Testing
Having tested the requirements, analyzed the design, and performed code review, it might be assumed that all issues have been
caught. Hopefully this is the case, but penetration testing the application after it has been deployed provides a last check to ensure
that nothing has been missed.
Phase 4.2: Configuration Management Testing
The application penetration test should include the checking of how
the infrastructure was deployed and secured. While the application
may be secure, a small aspect of the configuration could still be at a
default install stage and vulnerable to exploitation.
26
The OWASP Testing Framework
Phase 5: Maintenance and Operations
Phase 5.1: Conduct Operational Management Reviews
There needs to be a process in place which details how the operational side of both the application and infrastructure is managed.
Phase 5.2: Conduct Periodic Health Checks
Monthly or quarterly health checks should be performed on both
the application and infrastructure to ensure no new security risks
have been introduced and that the level of security is still intact.
Phase 5.3: Ensure Change Verification
After every change has been approved and tested in the QA environment and deployed into the production environment, it is vital
that the change is checked to ensure that the level of security has
not been affected by the change. This should be integrated into the
change management process.
A Typical SDLC Testing Workflow
The following figure shows a typical SDLC Testing Workflow.
OWASP TESTING FRAMEWORK WORK FLOW
Before
Development
Review SDLC
Process
Metrics
Criteria
Measurement
Traceability
Policy Review
Standards
Review
Requirements
Review
Design and
Architecture
Review
Create /
Review UML
models
Development
Code Review
Code
Walkthroughs
Unit and
System tests
Deployment
Penetration
Testing
Configuration
Management
Reviews
Unit and
System tests
Acceptance
Tests
Maintenance
Chance
verification
Health Checks
Operational
Management
reviews
Regression
Tests
Definition
and Design
Create /
Review Threat
Models
27
Web Application Penetration Testing
4
Web Application
Security Testing
The following sections describe the 12
subcategories of the Web Application
Penetration Testing Methodology:
Testing: Introduction and objectives
This section describes the OWASP web application security testing
methodology and explains how to test for evidence of vulnerabilities
within the application due to deficiencies with identified security controls.
What is Web Application Security Testing?
A security test is a method of evaluating the security of a computer
system or network by methodically validating and verifying the effectiveness of application security controls. A web application security
test focuses only on evaluating the security of a web application. The
process involves an active analysis of the application for any weaknesses, technical flaws, or vulnerabilities. Any security issues that are
found will be presented to the system owner, together with an assessment of the impact, a proposal for mitigation or a technical solution.
What is a Vulnerability?
A vulnerability is a flaw or weakness in a system’s design, implementation, operation or management that could be exploited to compromise the system’s security objectives.
What is a Threat?
A threat is anything (a malicious external attacker, an internal user, a
system instability, etc) that may harm the assets owned by an application (resources of value, such as the data in a database or in the file
system) by exploiting a vulnerability.
What is a Test?
A test is an action to demonstrate that an application meets the security requirements of its stakeholders.
The Approach in Writing this Guide
The OWASP approach is open and collaborative:
• Open: every security expert can participate with his or her experience
in the project. Everything is free.
• Collaborative: brainstorming is performed before the articles are
written so the team can share ideas and develop a collective vision
of the project. That means rough consensus, a wider audience and
increased participation.
This approach tends to create a defined Testing Methodology that
will be:
• Consistent
• Reproducible
• Rigorous
• Under quality control
The problems to be addressed are fully documented and tested. It is
important to use a method to test all known vulnerabilities and document all the security test activities.
What is the OWASP testing methodology?
Security testing will never be an exact science where a complete list
of all possible issues that should be tested can be defined. Indeed,
security testing is only an appropriate technique for testing the security of web applications under certain circumstances. The goal of this
project is to collect all the possible testing techniques, explain these
techniques, and keep the guide updated. The OWASP Web Application
Security Testing method is based on the black box approach. The tester knows nothing or has very little information about the application
to be tested.
The testing model consists of:
• Tester: Who performs the testing activities
• Tools and methodology: The core of this Testing Guide project
• Application: The black box to test
The test is divided into 2 phases:
• Phase 1 Passive mode:
In the passive mode the tester tries to understand the application’s
logic and plays with the application. Tools can be used for information gathering. For example, an HTTP proxy can be used to observe all
the HTTP requests and responses. At the end of this phase, the tester
should understand all the access points (gates) of the application (e.g.,
HTTP headers, parameters, and cookies). The Information Gathering
section explains how to perform a passive mode test.
For example the tester could find the following:
https://www.example.com/login/Authentic_Form.html
This may indicate an authentication form where the application requests a username and a password.
The following parameters represent two access points (gates) to the
application:
http://www.example.com/Appx.jsp?a=1&b=1
In this case, the application shows two gates (parameters a and b). All
the gates found in this phase represent a point of testing. A spreadsheet with the directory tree of the application and all the access
points would be useful for the second phase.
28
Web Application Penetration Testing
• Phase 2 Active mode:
In this phase the tester begins to test using the methodology described in the follow sections.
The set of active tests have been split into 11 sub-categories for a
total of 91 controls:
• Information Gathering
• Configuration and Deployment Management Testing
• Identity Management Testing
• Authentication Testing
• Authorization Testing
• Session Management Testing
• Input Validation Testing
• Error Handling
• Cryptography
• Business Logic Testing
• Client Side Testing
Testing for Information Gathering
Understanding the deployed configuration of the server hosting the
web application is almost as important as the application security testing itself. After all, an application chain is only as strong as its
weakest link. Application platforms are wide and varied, but some key
platform configuration errors can compromise the application in the
same way an unsecured application can compromise the server.
Conduct search engine discovery/reconnaissance
for information leakage (OTG-INFO-001)
Summary
There are direct and indirect elements to search engine discovery
and reconnaissance. Direct methods relate to searching the indexes
and the associated content from caches. Indirect methods relate to
gleaning sensitive design and configuration information by searching
forums, newsgroups, and tendering websites.
Once a search engine robot has completed crawling, it commences indexing the web page based on tags and associated attributes, such as
, in order to return the relevant search results [1]. If the robots.
txt file is not updated during the lifetime of the web site, and inline
HTML meta tags that instruct robots not to index content have not
been used, then it is possible for indexes to contain web content not
intended to be included in by the owners. Website owners may use
the previously mentioned robots.txt, HTML meta tags, authentication,
and tools provided by search engines to remove such content.
Search operators
Using the advanced “site:” search operator, it is possible to restrict
search results to a specific domain [2]. Do not limit testing to just
one search engine provider as they may generate different results
depending on when they crawled content and their own algorithms.
Consider using the following search engines:
• Baidu
• binsearch.info
• Bing
• Duck Duck Go
• ixquick/Startpage
• Google
• Shodan
• PunkSpider
Duck Duck Go and ixquick/Startpage provide reduced information
leakage about the tester.
Google provides the Advanced “cache:” search operator [2], but this
is the equivalent to clicking the “Cached” next to each Google Search
Result. Hence, the use of the Advanced “site:” Search Operator and
then clicking “Cached” is preferred.
The Google SOAP Search API supports the doGetCachedPage and the
associated doGetCachedPageResponse SOAP Messages [3] to assist
with retrieving cached pages. An implementation of this is under development by the OWASP “Google Hacking” Project.
PunkSpider is web application vulnerability search engine. It is of little use for a penetration tester doing manual work. However it can
be useful as demonstration of easiness of finding vulnerabilities by
script-kiddies.
Example To find the web content of owasp.org indexed by a typical
search engine, the syntax required is:
site:owasp.org
Test Objectives
To understand what sensitive design and configuration information of
the application/system/organization is exposed both directly (on the
organization’s website) or indirectly (on a third party website).
How to Test
Use a search engine to search for:
• Network diagrams and configurations
• Archived posts and emails by administrators and other key staff
• Log on procedures and username formats
• Usernames and passwords
• Error message content
• Development, test, UAT and staging versions of the website
To display the index.html of owasp.org as cached, the syntax is:
cache:owasp.org
29
Web Application Penetration Testing
Google Hacking Database
The Google Hacking Database is list of useful search queries for Google. Queries are put in several categories:
• Footholds
• Files containing usernames
• Sensitive Directories
• Web Server Detection
• Vulnerable Files
• Vulnerable Servers
• Error Messages
• Files containing juicy info
• Files containing passwords
• Sensitive Online Shopping Info
Tools
[4] FoundStone SiteDigger: http://www.mcafee.com/uk/downloads/
free-tools/sitedigger.aspx
[5] Google Hacker: http://yehg.net/lab/pr0js/files.php/googlehacker.
zip
[6] Stach & Liu’s Google Hacking Diggity Project: http://www.stachliu.com/resources/tools/google-hacking-diggity-project/
[7] PunkSPIDER: http://punkspider.hyperiongray.com/
References
Web
[1] “Google Basics: Learn how Google Discovers, Crawls, and
Serves Web Pages” - https://support.google.com/webmasters/answer/70897
[2] “Operators and More Search Help”: https://support.google.com/
websearch/answer/136861?hl=en
[3] “Google Hacking Database”: http://www.exploit-db.com/google-dorks/
Remediation
Carefully consider the sensitivity of design and configuration information before it is posted online.
Periodically review the sensitivity of existing design and configuration
information that is posted online.
Fingerprint Web Server (OTG-INFO-002)
Summary
Web server fingerprinting is a critical task for the penetration tester.
Knowing the version and type of a running web server allows testers
to determine known vulnerabilities and the appropriate exploits to use
during testing.
There are several different vendors and versions of web servers on
the market today. Knowing the type of web server that is being tested significantly helps in the testing process and can also change the
course of the test.
This information can be derived by sending the web server specific
commands and analyzing the output, as each version of web server
software may respond differently to these commands. By knowing
how each type of web server responds to specific commands and
keeping this information in a web server fingerprint database, a penetration tester can send these commands to the web server, analyze
the response, and compare it to the database of known signatures.
Please note that it usually takes several different commands to accurately identify the web server, as different versions may react similarly
to the same command. Rarely do different versions react the same to
all HTTP commands. So by sending several different commands, the
tester can increase the accuracy of their guess.
Test Objectives
Find the version and type of a running web server to determine known
vulnerabilities and the appropriate exploits to use during testing.
How to Test
Black Box testing
The simplest and most basic form of identifying a web server is to look
at the Server field in the HTTP response header. Netcat is used in this
experiment.
Consider the following HTTP Request-Response:
$ nc 202.41.76.251 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Date: Mon, 16 Jun 2003 02:53:29 GMT
Server: Apache/1.3.3 (Unix) (Red Hat/Linux)
Last-Modified: Wed, 07 Oct 1998 11:18:14 GMT
ETag: “1813-49b-361b4df6”
Accept-Ranges: bytes
Content-Length: 1179
Connection: close
Content-Type: text/html
From the Server field, one can understand that the server is likely
Apache, version 1.3.3, running on Linux operating system.
Four examples of the HTTP response headers are shown below.
From an Apache 1.3.23 server:
HTTP/1.1 200 OK
Date: Sun, 15 Jun 2003 17:10: 49 GMT
Server: Apache/1.3.23
Last-Modified: Thu, 27 Feb 2003 03:48: 19 GMT
ETag: 32417-c4-3e5d8a83
Accept-Ranges: bytes
Content-Length: 196
Connection: close
Content-Type: text/HTML
30
Web Application Penetration Testing
From a Microsoft IIS 5.0 server:
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Expires: Yours, 17 Jun 2003 01:41: 33 GMT
Date: Mon, 16 Jun 2003 01:41: 33 GMT
Content-Type: text/HTML
Accept-Ranges: bytes
Last-Modified: Wed, 28 May 2003 15:32: 21 GMT
ETag: b0aac0542e25c31: 89d
Content-Length: 7369
From a Netscape Enterprise 4.1 server:
HTTP/1.1 200 OK
Server: Netscape-Enterprise/4.1
Date: Mon, 16 Jun 2003 06:19: 04 GMT
Content-type: text/HTML
Last-modified: Wed, 31 Jul 2002 15:37: 56 GMT
Content-length: 57
Accept-ranges: bytes
Connection: close
From a SunONE 6.1 server:
HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Tue, 16 Jan 2007 14:53:45 GMT
Content-length: 1186
Content-type: text/html
Date: Tue, 16 Jan 2007 14:50:31 GMT
Last-Modified: Wed, 10 Jan 2007 09:58:26 GMT
Accept-Ranges: bytes
Connection: close
However, this testing methodology is limited in accuracy. There are
several techniques that allow a web site to obfuscate or to modify the
server banner string. For example one could obtain the following answer:
403 HTTP/1.1 Forbidden
Date: Mon, 16 Jun 2003 02:41: 27 GMT
Server: Unknown-Webserver/1.0
Connection: close
Content-Type: text/HTML; charset=iso-8859-1
In this case, the server field of that response is obfuscated. The tester cannot know what type of web server is running based on such
information.
Protocol Behavior
More refined techniques take in consideration various characteristics
of the several web servers available on the market. Below is a list of
some methodologies that allow testers to deduce the type of web
server in use.
HTTP header field ordering
The first method consists of observing the ordering of the several
headers in the response. Every web server has an inner ordering of the
header. Consider the following answers as an example:
Response from Apache 1.3.23
$ nc apache.example.com 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Date: Sun, 15 Jun 2003 17:10: 49 GMT
Server: Apache/1.3.23
Last-Modified: Thu, 27 Feb 2003 03:48: 19 GMT
ETag: 32417-c4-3e5d8a83
Accept-Ranges: bytes
Content-Length: 196
Connection: close
Content-Type: text/HTML
Response from IIS 5.0
$ nc iis.example.com 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Content-Location: http://iis.example.com/Default.htm
Date: Fri, 01 Jan 1999 20:13: 52 GMT
Content-Type: text/HTML
Accept-Ranges: bytes
Last-Modified: Fri, 01 Jan 1999 20:13: 52 GMT
ETag: W/e0d362a4c335be1: ae1
Content-Length: 133
Response from Netscape Enterprise 4.1
$ nc netscape.example.com 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Server: Netscape-Enterprise/4.1
Date: Mon, 16 Jun 2003 06:01: 40 GMT
Content-type: text/HTML
Last-modified: Wed, 31 Jul 2002 15:37: 56 GMT
Content-length: 57
Accept-ranges: bytes
Connection: close
31
Web Application Penetration Testing
Response from a SunONE 6.1
$ nc sunone.example.com 80
HEAD / HTTP/1.0
HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Tue, 16 Jan 2007 15:23:37 GMT
Content-length: 0
Content-type: text/html
Date: Tue, 16 Jan 2007 15:20:26 GMT
Last-Modified: Wed, 10 Jan 2007 09:58:26 GMT
Connection: close
We can notice that the ordering of the Date field and the Server field
differs between Apache, Netscape Enterprise, and IIS.
Malformed requests test
Another useful test to execute involves sending malformed requests
or requests of nonexistent pages to the server. Consider the following
HTTP responses.
Response from Apache 1.3.23
$ nc apache.example.com 80
GET / HTTP/3.0
HTTP/1.1 400 Bad Request
Date: Sun, 15 Jun 2003 17:12: 37 GMT
Server: Apache/1.3.23
Connection: close
Transfer: chunked
Content-Type: text/HTML; charset=iso-8859-1
Response from IIS 5.0
$ nc iis.example.com 80
GET / HTTP/3.0
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.0
Content-Location: http://iis.example.com/Default.htm
Date: Fri, 01 Jan 1999 20:14: 02 GMT
Content-Type: text/HTML
Accept-Ranges: bytes
Last-Modified: Fri, 01 Jan 1999 20:14: 02 GMT
ETag: W/e0d362a4c335be1: ae1
Content-Length: 133
Response from Netscape Enterprise 4.1
$ nc netscape.example.com 80
GET / HTTP/3.0
HTTP/1.1 505 HTTP Version Not Supported
Server: Netscape-Enterprise/4.1
Date: Mon, 16 Jun 2003 06:04: 04 GMT
Content-length: 140
Content-type: text/HTML
Connection: close
Response from a SunONE 6.1
$ nc sunone.example.com 80
GET / HTTP/3.0
HTTP/1.1 400 Bad request
Server: Sun-ONE-Web-Server/6.1
Date: Tue, 16 Jan 2007 15:25:00 GMT
Content-length: 0
Content-type: text/html
Connection: close
We notice that every server answers in a different way. The answer
also differs in the version of the server. Similar observations can be
done we create requests with a non-existent HTTP method/verb.
Consider the following responses:
Response from Apache 1.3.23
$ nc apache.example.com 80
GET / JUNK/1.0
HTTP/1.1 200 OK
Date: Sun, 15 Jun 2003 17:17: 47 GMT
Server: Apache/1.3.23
Last-Modified: Thu, 27 Feb 2003 03:48: 19 GMT
ETag: 32417-c4-3e5d8a83
Accept-Ranges: bytes
Content-Length: 196
Connection: close
Content-Type: text/HTML
Response from IIS 5.0
$ nc iis.example.com 80
GET / JUNK/1.0
HTTP/1.1 400 Bad Request
Server: Microsoft-IIS/5.0
Date: Fri, 01 Jan 1999 20:14: 34 GMT
Content-Type: text/HTML
Content-Length: 87
32
Web Application Penetration Testing
Response from Netscape Enterprise 4.1
$ nc netscape.example.com 80
GET / JUNK/1.0
of an online tool that often delivers a lot of information about target
Web Servers, is Netcraft. With this tool we can retrieve information
about operating system, web server used, Server Uptime, Netblock
Owner, history of change related to Web server and O.S.
An example is shown below:
<HTML><HEAD><TITLE>Bad request
Bad request
Your browser sent to query this server could not understand.