Programming Python OReilly 4th Ed. Ed
User Manual: OReilly-Programming-Python-4th-ed.
Open the PDF directly: View PDF 
.
Page Count: 1628
| Download | |
| Open PDF In Browser | View PDF |
Programming Python FOURTH EDITION Programming Python Mark Lutz Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo Programming Python, Fourth Edition by Mark Lutz Copyright © 2011 Mark Lutz. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Julie Steele Production Editor: Teresa Elsey Proofreader: Teresa Elsey Indexer: Lucie Haskins Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Robert Romano Printing History: October 1996: March 2001: August 2006: December 2010: First Edition. Second Edition. Third Edition. Fourth Edition. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Programming Python, the image of an African rock python, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-0-596-15810-1 [QG] 1292258056 Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii Part I. The Beginning 1. A Sneak Preview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 “Programming Python: The Short Story” The Task Step 1: Representing Records Using Lists Using Dictionaries Step 2: Storing Records Persistently Using Formatted Files Using Pickle Files Using Per-Record Pickle Files Using Shelves Step 3: Stepping Up to OOP Using Classes Adding Behavior Adding Inheritance Refactoring Code Adding Persistence Other Database Options Step 4: Adding Console Interaction A Console Shelve Interface Step 5: Adding a GUI GUI Basics Using OOP for GUIs Getting Input from a User A GUI Shelve Interface Step 6: Adding a Web Interface CGI Basics 3 4 4 4 9 14 14 19 22 23 26 27 29 29 31 34 36 37 37 40 40 42 44 46 52 52 v Running a Web Server Using Query Strings and urllib Formatting Reply Text A Web-Based Shelve Interface The End of the Demo 55 57 59 60 69 Part II. System Programming 2. System Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 “The os.path to Knowledge” Why Python Here? The Next Five Chapters System Scripting Overview Python System Modules Module Documentation Sources Paging Documentation Strings A Custom Paging Script String Method Basics Other String Concepts in Python 3.X: Unicode and bytes File Operation Basics Using Programs in Two Ways Python Library Manuals Commercially Published References Introducing the sys Module Platforms and Versions The Module Search Path The Loaded Modules Table Exception Details Other sys Module Exports Introducing the os Module Tools in the os Module Administrative Tools Portability Constants Common os.path Tools Running Shell Commands from Scripts Other os Module Exports 73 73 74 75 76 77 78 79 80 82 83 84 85 86 86 86 87 88 89 90 90 90 91 92 92 94 100 3. Script Execution Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 “I’d Like to Have an Argument, Please” Current Working Directory CWD, Files, and Import Paths CWD and Command Lines vi | Table of Contents 103 104 104 106 Command-Line Arguments Parsing Command-Line Arguments Shell Environment Variables Fetching Shell Variables Changing Shell Variables Shell Variable Fine Points: Parents, putenv, and getenv Standard Streams Redirecting Streams to Files and Programs Redirected Streams and User Interaction Redirecting Streams to Python Objects The io.StringIO and io.BytesIO Utility Classes Capturing the stderr Stream Redirection Syntax in Print Calls Other Redirection Options: os.popen and subprocess Revisited 106 107 109 110 111 112 113 114 119 123 126 127 127 128 4. File and Directory Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 “Erase Your Hard Drive in Five Easy Steps!” File Tools The File Object Model in Python 3.X Using Built-in File Objects Binary and Text Files Lower-Level File Tools in the os Module File Scanners Directory Tools Walking One Directory Walking Directory Trees Handling Unicode Filenames in 3.X: listdir, walk, glob 135 135 136 137 146 155 160 163 164 168 172 5. Parallel System Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 “Telling the Monkeys What to Do” Forking Processes The fork/exec Combination Threads The _thread Module The threading Module The queue Module Preview: GUIs and Threads More on the Global Interpreter Lock Program Exits sys Module Exits os Module Exits Shell Command Exit Status Codes Process Exit Status and Shared State 177 179 182 186 189 199 204 208 211 213 214 215 216 219 Table of Contents | vii Thread Exits and Shared State Interprocess Communication Anonymous Pipes Named Pipes (Fifos) Sockets: A First Look Signals The multiprocessing Module Why multiprocessing? The Basics: Processes and Locks IPC Tools: Pipes, Shared Memory, and Queues Starting Independent Programs And Much More Why multiprocessing? The Conclusion Other Ways to Start Programs The os.spawn Calls The os.startfile call on Windows A Portable Program-Launch Framework Other System Tools Coverage 220 222 224 234 236 240 243 243 245 248 254 256 257 258 258 261 263 268 6. Complete System Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 “The Greps of Wrath” A Quick Game of “Find the Biggest Python File” Scanning the Standard Library Directory Scanning the Standard Library Tree Scanning the Module Search Path Scanning the Entire Machine Printing Unicode Filenames Splitting and Joining Files Splitting Files Portably Joining Files Portably Usage Variations Generating Redirection Web Pages Page Template File Page Generator Script A Regression Test Script Running the Test Driver Copying Directory Trees Comparing Directory Trees Finding Directory Differences Finding Tree Differences Running the Script Verifying Backups Reporting Differences and Other Ideas viii | Table of Contents 271 272 272 273 274 276 279 282 283 286 289 292 293 294 297 299 304 308 309 311 314 316 317 Searching Directory Trees Greps and Globs and Finds Rolling Your Own find Module Cleaning Up Bytecode Files A Python Tree Searcher Visitor: Walking Directories “++” Editing Files in Directory Trees (Visitor) Global Replacements in Directory Trees (Visitor) Counting Source Code Lines (Visitor) Recoding Copies with Classes (Visitor) Other Visitor Examples (External) Playing Media Files The Python webbrowser Module The Python mimetypes Module Running the Script Automated Program Launchers (External) 319 320 321 324 327 330 334 336 338 339 341 343 347 348 350 351 Part III. GUI Programming 7. Graphical User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 “Here’s Looking at You, Kid” GUI Programming Topics Running the Examples Python GUI Development Options tkinter Overview tkinter Pragmatics tkinter Documentation tkinter Extensions tkinter Structure Climbing the GUI Learning Curve “Hello World” in Four Lines (or Less) tkinter Coding Basics Making Widgets Geometry Managers Running GUI Programs tkinter Coding Alternatives Widget Resizing Basics Configuring Widget Options and Window Titles One More for Old Times’ Sake Packing Widgets Without Saving Them Adding Buttons and Callbacks Widget Resizing Revisited: Expansion 355 355 357 358 363 363 364 364 366 368 368 369 370 370 371 372 373 375 376 377 379 380 Table of Contents | ix Adding User-Defined Callback Handlers Lambda Callback Handlers Deferring Calls with Lambdas and Object References Callback Scope Issues Bound Method Callback Handlers Callable Class Object Callback Handlers Other tkinter Callback Protocols Binding Events Adding Multiple Widgets Widget Resizing Revisited: Clipping Attaching Widgets to Frames Layout: Packing Order and Side Attachments The Packer’s Expand and Fill Revisited Using Anchor to Position Instead of Stretch Customizing Widgets with Classes Standardizing Behavior and Appearance Reusable GUI Components with Classes Attaching Class Components Extending Class Components Standalone Container Classes The End of the Tutorial Python/tkinter for Tcl/Tk Converts 382 383 384 385 391 392 393 394 395 396 397 397 398 399 400 401 403 405 407 408 410 412 8. A tkinter Tour, Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 “Widgets and Gadgets and GUIs, Oh My!” This Chapter’s Topics Configuring Widget Appearance Top-Level Windows Toplevel and Tk Widgets Top-Level Window Protocols Dialogs Standard (Common) Dialogs The Old-Style Dialog Module Custom Dialogs Binding Events Other bind Events Message and Entry Message Entry Laying Out Input Forms tkinter “Variables” and Form Layout Alternatives Checkbutton, Radiobutton, and Scale Checkbuttons x | Table of Contents 415 415 416 419 421 422 426 426 438 439 443 447 448 448 449 451 454 457 457 Radio Buttons Scales (Sliders) Running GUI Code Three Ways Attaching Frames Independent Windows Running Programs Images Fun with Buttons and Pictures Viewing and Processing Images with PIL PIL Basics Displaying Other Image Types with PIL Creating Image Thumbnails with PIL 462 467 471 471 476 478 484 487 491 491 493 496 9. A tkinter Tour, Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 “On Today’s Menu: Spam, Spam, and Spam” Menus Top-Level Window Menus Frame- and Menubutton-Based Menus Windows with Both Menus and Toolbars Listboxes and Scrollbars Programming Listboxes Programming Scroll Bars Packing Scroll Bars Text Programming the Text Widget Adding Text-Editing Operations Unicode and the Text Widget Advanced Text and Tag Operations Canvas Basic Canvas Operations Programming the Canvas Widget Scrolling Canvases Scrollable Canvases and Image Thumbnails Using Canvas Events Grids Why Grids? Grid Basics: Input Forms Revisited Comparing grid and pack Combining grid and pack Making Gridded Widgets Expandable Laying Out Larger Tables with grid Time Tools, Threads, and Animation Using Threads with tkinter GUIs 507 507 508 512 517 522 524 525 526 528 530 533 538 548 550 550 551 554 557 560 564 564 565 566 568 570 574 582 584 Table of Contents | xi Using the after Method Simple Animation Techniques Other Animation Topics The End of the Tour Other Widgets and Options 585 588 593 595 595 10. GUI Coding Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 “Building a Better Mousetrap” GuiMixin: Common Tool Mixin Classes Widget Builder Functions Mixin Utility Classes GuiMaker: Automating Menus and Toolbars Subclass Protocols GuiMaker Classes GuiMaker Self-Test BigGui: A Client Demo Program ShellGui: GUIs for Command-Line Tools A Generic Shell-Tools Display Application-Specific Tool Set Classes Adding GUI Frontends to Command Lines GuiStreams: Redirecting Streams to Widgets Using Redirection for the Packing Scripts Reloading Callback Handlers Dynamically Wrapping Up Top-Level Window Interfaces GUIs, Threads, and Queues Placing Data on Queues Placing Callbacks on Queues More Ways to Add GUIs to Non-GUI Code Popping Up GUI Windows on Demand Adding a GUI As a Separate Program: Sockets (A Second Look) Adding a GUI As a Separate Program: Command Pipes The PyDemos and PyGadgets Launchers PyDemos Launcher Bar (Mostly External) PyGadgets Launcher Bar 597 598 598 599 603 607 608 608 609 613 613 615 617 623 627 628 630 635 636 640 646 647 649 654 662 662 667 11. Complete GUI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 671 “Python, Open Source, and Camaros” Examples in Other Chapters This Chapter’s Strategy PyEdit: A Text Editor Program/Object Running PyEdit PyEdit Changes in Version 2.0 (Third Edition) PyEdit Changes in Version 2.1 (Fourth Edition) xii | Table of Contents 671 672 673 674 675 682 684 PyEdit Source Code PyPhoto: An Image Viewer and Resizer Running PyPhoto PyPhoto Source Code PyView: An Image and Notes Slideshow Running PyView PyView Source Code PyDraw: Painting and Moving Graphics Running PyDraw PyDraw Source Code PyClock: An Analog/Digital Clock Widget A Quick Geometry Lesson Running PyClock PyClock Source Code PyToe: A Tic-Tac-Toe Game Widget Running PyToe PyToe Source Code (External) Where to Go from Here 693 716 717 719 727 727 732 738 738 738 747 747 751 754 762 762 763 766 Part IV. Internet Programming 12. Network Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 771 “Tune In, Log On, and Drop Out” Internet Scripting Topics Running Examples in This Part of the Book Python Internet Development Options Plumbing the Internet The Socket Layer The Protocol Layer Python’s Internet Library Modules Socket Programming Socket Basics Running Socket Programs Locally Running Socket Programs Remotely Spawning Clients in Parallel Talking to Reserved Ports Handling Multiple Clients Forking Servers Threading Servers Standard Library Server Classes Multiplexing Servers with select Summary: Choosing a Server Scheme 771 772 775 777 780 781 782 785 787 788 794 795 798 801 802 803 815 818 820 826 Table of Contents | xiii Making Sockets Look Like Files and Streams A Stream Redirection Utility A Simple Python File Server Running the File Server and Clients Adding a User-Interface Frontend 827 828 840 842 843 13. Client-Side Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 853 “Socket to Me!” FTP: Transferring Files over the Net Transferring Files with ftplib Using urllib to Download Files FTP get and put Utilities Adding a User Interface Transferring Directories with ftplib Downloading Site Directories Uploading Site Directories Refactoring Uploads and Downloads for Reuse Transferring Directory Trees with ftplib Uploading Local Trees Deleting Remote Trees Downloading Remote Trees Processing Internet Email Unicode in Python 3.X and Email Tools POP: Fetching Email Mail Configuration Module POP Mail Reader Script Fetching Messages Fetching Email at the Interactive Prompt SMTP: Sending Email SMTP Mail Sender Script Sending Messages Sending Email at the Interactive Prompt email: Parsing and Composing Mail Content Message Objects Basic email Package Interfaces in Action Unicode, Internationalization, and the Python 3.1 email Package A Console-Based Email Client Running the pymail Console Client The mailtools Utility Package Initialization File MailTool Class MailSender Class MailFetcher Class xiv | Table of Contents 853 854 854 857 860 867 874 874 880 884 892 893 895 899 899 900 901 902 905 906 909 910 911 913 919 921 922 924 926 947 952 956 957 958 959 967 MailParser Class Self-Test Script Updating the pymail Console Client NNTP: Accessing Newsgroups HTTP: Accessing Websites The urllib Package Revisited Other urllib Interfaces Other Client-Side Scripting Options 976 983 986 991 994 997 999 1002 14. The PyMailGUI Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005 “Use the Source, Luke” Source Code Modules and Size Why PyMailGUI? Running PyMailGUI Presentation Strategy Major PyMailGUI Changes New in Version 2.1 and 2.0 (Third Edition) New in Version 3.0 (Fourth Edition) A PyMailGUI Demo Getting Started Loading Mail Threading Model Load Server Interface Offline Processing with Save and Open Sending Email and Attachments Viewing Email and Attachments Email Replies and Forwards and Recipient Options Deleting Email POP Message Numbers and Synchronization Handling HTML Content in Email Mail Content Internationalization Support Alternative Configurations and Accounts Multiple Windows and Status Messages PyMailGUI Implementation PyMailGUI: The Main Module SharedNames: Program-Wide Globals ListWindows: Message List Windows ViewWindows: Message View Windows messagecache: Message Cache Manager popuputil: General-Purpose GUI Pop Ups wraplines: Line Split Tools html2text: Extracting Text from HTML (Prototype, Preview) mailconfig: User Configurations 1005 1006 1008 1010 1010 1011 1011 1012 1019 1020 1025 1027 1030 1031 1033 1037 1043 1049 1051 1053 1055 1059 1060 1062 1063 1066 1067 1085 1095 1098 1100 1102 1105 Table of Contents | xv textConfig: Customizing Pop-Up PyEdit Windows PyMailGUIHelp: User Help Text and Display altconfigs: Configuring for Multiple Accounts Ideas for Improvement 1110 1111 1114 1116 15. Server-Side Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1125 “Oh, What a Tangled Web We Weave” What’s a Server-Side CGI Script? The Script Behind the Curtain Writing CGI Scripts in Python Running Server-Side Examples Web Server Options Running a Local Web Server The Server-Side Examples Root Page Viewing Server-Side Examples and Output Climbing the CGI Learning Curve A First Web Page A First CGI Script Adding Pictures and Generating Tables Adding User Interaction Using Tables to Lay Out Forms Adding Common Input Devices Changing Input Layouts Passing Parameters in Hardcoded URLs Passing Parameters in Hidden Form Fields Saving State Information in CGI Scripts URL Query Parameters Hidden Form Input Fields HTTP “Cookies” Server-Side Databases Extensions to the CGI Model Combining Techniques The Hello World Selector Checking for Missing and Invalid Inputs Refactoring Code for Maintainability Step 1: Sharing Objects Between Pages—A New Input Form Step 2: A Reusable Form Mock-Up Utility Step 3: Putting It All Together—A New Reply Script More on HTML and URL Escapes URL Escape Code Conventions Python HTML and URL Escape Tools Escaping HTML Code Escaping URLs xvi | Table of Contents 1125 1126 1126 1128 1130 1130 1131 1133 1134 1135 1135 1141 1146 1149 1157 1163 1166 1170 1172 1174 1176 1176 1177 1181 1182 1183 1183 1190 1192 1193 1196 1199 1201 1202 1203 1203 1204 Escaping URLs Embedded in HTML Code Transferring Files to Clients and Servers Displaying Arbitrary Server Files on the Client Uploading Client Files to the Server More Than One Way to Push Bits over the Net 1205 1209 1211 1218 1227 16. The PyMailCGI Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1229 “Things to Do When Visiting Chicago” The PyMailCGI Website Implementation Overview New in This Fourth Edition (Version 3.0) New in the Prior Edition (Version 2.0) Presentation Overview Running This Chapter’s Examples The Root Page Configuring PyMailCGI Sending Mail by SMTP The Message Composition Page The Send Mail Script Error Pages Common Look-and-Feel Using the Send Mail Script Outside a Browser Reading POP Email The POP Password Page The Mail Selection List Page Passing State Information in URL Link Parameters Security Protocols The Message View Page Passing State Information in HTML Hidden Input Fields Escaping Mail Text and Passwords in HTML Processing Fetched Mail Reply and Forward Delete Deletions and POP Message Numbers Utility Modules External Components and Configuration POP Mail Interface POP Password Encryption Common Utilities Module Web Scripting Trade-Offs PyMailCGI Versus PyMailGUI The Web Versus the Desktop Other Approaches 1229 1230 1230 1233 1235 1236 1237 1239 1240 1241 1242 1242 1246 1246 1247 1249 1250 1251 1254 1257 1259 1262 1264 1266 1267 1268 1272 1276 1276 1277 1278 1286 1291 1292 1293 1296 Table of Contents | xvii Part V. Tools and Techniques 17. Databases and Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1303 “Give Me an Order of Persistence, but Hold the Pickles” Persistence Options in Python DBM Files Using DBM Files DBM Details: Files, Portability, and Close Pickled Objects Using Object Pickling Pickling in Action Pickle Details: Protocols, Binary Modes, and _pickle Shelve Files Using Shelves Storing Built-in Object Types in Shelves Storing Class Instances in Shelves Changing Classes of Objects Stored in Shelves Shelve Constraints Pickled Class Constraints Other Shelve Limitations The ZODB Object-Oriented Database The Mostly Missing ZODB Tutorial SQL Database Interfaces SQL Interface Overview An SQL Database API Tutorial with SQLite Building Record Dictionaries Tying the Pieces Together Loading Database Tables from Files SQL Utility Scripts SQL Resources ORMs: Object Relational Mappers PyForm: A Persistent Object Viewer (External) 1303 1303 1305 1305 1308 1309 1310 1311 1314 1315 1316 1317 1318 1320 1321 1323 1324 1325 1326 1329 1330 1332 1339 1342 1344 1347 1354 1354 1356 18. Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1359 “Roses Are Red, Violets Are Blue; Lists Are Mutable, and So Is Set Foo” Implementing Stacks Built-in Options A Stack Module A Stack Class Customization: Performance Monitors Optimization: Tuple Tree Stacks xviii | Table of Contents 1359 1360 1360 1362 1364 1366 1367 Optimization: In-Place List Modifications Timing the Improvements Implementing Sets Built-in Options Set Functions Set Classes Optimization: Moving Sets to Dictionaries Adding Relational Algebra to Sets (External) Subclassing Built-in Types Binary Search Trees Built-in Options Implementing Binary Trees Trees with Both Keys and Values Graph Searching Implementing Graph Search Moving Graphs to Classes Permuting Sequences Reversing and Sorting Sequences Implementing Reversals Implementing Sorts Data Structures Versus Built-ins: The Conclusion PyTree: A Generic Tree Object Viewer 1369 1371 1373 1374 1375 1377 1378 1382 1383 1385 1385 1386 1388 1390 1390 1393 1395 1397 1398 1399 1400 1402 19. Text and Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1405 “See Jack Hack. Hack, Jack, Hack” Strategies for Processing Text in Python String Method Utilities Templating with Replacements and Formats Parsing with Splits and Joins Summing Columns in a File Parsing and Unparsing Rule Strings Regular Expression Pattern Matching The re Module First Examples String Operations Versus Patterns Using the re Module More Pattern Examples Scanning C Header Files for Patterns XML and HTML Parsing XML Parsing in Action HTML Parsing in Action Advanced Language Tools Custom Language Parsers 1405 1405 1406 1408 1409 1410 1412 1415 1416 1416 1418 1421 1425 1427 1429 1430 1435 1438 1440 Table of Contents | xix The Expression Grammar The Parser’s Code Adding a Parse Tree Interpreter Parse Tree Structure Exploring Parse Trees with the PyTree GUI Parsers Versus Python PyCalc: A Calculator Program/Object A Simple Calculator GUI PyCalc—A “Real” Calculator GUI 1440 1441 1449 1454 1456 1457 1457 1458 1463 20. Python/C Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1483 “I Am Lost at C” Extending and Embedding Extending Python in C: Overview A Simple C Extension Module The SWIG Integration Code Generator A Simple SWIG Example Wrapping C Environment Calls Adding Wrapper Classes to Flat Libraries Wrapping C Environment Calls with SWIG Wrapping C++ Classes with SWIG A Simple C++ Extension Class Wrapping the C++ Class with SWIG Using the C++ Class in Python Other Extending Tools Embedding Python in C: Overview The C Embedding API What Is Embedded Code? Basic Embedding Techniques Running Simple Code Strings Running Code Strings with Results and Namespaces Calling Python Objects Running Strings in Dictionaries Precompiling Strings to Bytecode Registering Callback Handler Objects Registration Implementation Using Python Classes in C Other Integration Topics xx | Table of Contents 1483 1484 1486 1487 1491 1491 1495 1499 1500 1502 1503 1505 1507 1511 1514 1515 1516 1518 1519 1522 1524 1526 1528 1530 1531 1535 1538 Part VI. The End 21. Conclusion: Python and the Development Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 1543 “That’s the End of the Book, Now Here’s the Meaning of Life” “Something’s Wrong with the Way We Program Computers” The “Gilligan Factor” Doing the Right Thing The Static Language Build Cycle Artificial Complexities One Language Does Not Fit All Enter Python But What About That Bottleneck? Python Provides Immediate Turnaround Python Is “Executable Pseudocode” Python Is OOP Done Right Python Fosters Hybrid Applications On Sinking the Titanic So What’s “Python: The Sequel”? In the Final Analysis… 1544 1544 1544 1545 1546 1546 1546 1547 1548 1549 1550 1550 1551 1552 1555 1555 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1557 Table of Contents | xxi Preface “And Now for Something Completely Different…” This book explores ways to apply the Python programming language in common application domains and realistically scaled tasks. It’s about what you can do with the language once you’ve mastered its fundamentals. This book assumes you are relatively new to each of the application domains it covers— GUIs, the Internet, databases, systems programming, and so on—and presents each from the ground up, in tutorial fashion. Along the way, it focuses on commonly used tools and libraries, rather than language fundamentals. The net result is a resource that provides readers with an in-depth understanding of Python’s roles in practical, realworld programming work. As a subtheme, this book also explores Python’s relevance as a software development tool—a role that many would classify as well beyond those typically associated with “scripting.” In fact, many of this book’s examples are scaled specifically for this purpose; among these, we’ll incrementally develop email clients that top out at thousands of lines of code. Programming at this full scale will always be challenging work, but we’ll find that it’s also substantially quicker and easier when done with Python. This Fourth Edition has been updated to present the language, libraries, and practice of Python 3.X. Specifically, its examples use Python 3.1—the most recent version of Python at the time of writing—and its major examples were tested successfully under the third alpha release of Python 3.2 just prior to publication, but they reflect the version of the language common to the entire 3.X line. This edition has also been reorganized in ways that both streamline some of its former material and allow for coverage of newly emerged tools and topics. Because this edition’s readership will include both newcomers as well as prior edition veterans, I want to use this Preface to expand on this book’s purpose and scope before we jump into code. xxiii About This Book This book is a tutorial introduction to using Python in common application domains and tasks. It teaches how to apply Python for system administration, GUIs, and the Web, and explores its roles in networking, databases, frontend scripting layers, text processing, and more. Although the Python language is used along the way, this book’s focus is on application to real-world tasks instead of language fundamentals. This Book’s Ecosystem Because of its scope, this book is designed to work best as the second of a two-volume set, and to be supplemented by a third. Most importantly, this book is an applications programming follow-up to the core language book Learning Python, whose subjects are officially prerequisite material here. Here’s how the three books are related: • Learning Python covers the fundamentals of Python programming in depth. It focuses on the core Python language, and its topics are prerequisite to this book. • Programming Python, this book, covers the application of Python to real-world programming tasks. It focuses on libraries and tools, and it assumes you already know Python fundamentals. • Python Pocket Reference provides a quick reference to details not listed exhaustively here. It doesn’t teach much, but it allows you to look up details fast. In some sense, this book is to application programming what Learning Python is to the core language—a gradual tutorial, which makes almost no assumptions about your background and presents each topic from the ground up. By studying this book’s coverage of Web basics, for example, you’ll be equipped to build simple websites, and you will be able to make sense of more advanced frameworks and tools as your needs evolve. GUIs are similarly taught incrementally, from basic to advanced. In addition, this book is designed to be supplemented by the quick-reference book Python Pocket Reference, which provides the small details finessed here and serves as a resource for looking up the fine points. That book is reference only, and is largely void of both examples and narrative, but it serves to augment and complement both Learning Python’s fundamentals and Programming Python’s applications. Because its current Fourth Edition gives both Python 2.X and 3.X versions of the tools it covers, that book also serves as a resource for readers transitioning between the two Python lines (more on this in a moment).* * Disclosure: I am the author of all three books mentioned in this section, which affords me the luxury of tightly controlling their scopes in order to avoid overlap. It also means that as an author, I try to avoid commenting on the many other Python books available, some of which are very good and may cover topics not addressed in any of my own books. Please see the Web for other Python resources. All three of my books reflect my 13 years on the Python training trail and stem from the original Programming Python written back in 1995. xxiv | Preface What This Book Is Not Because of the scopes carved out by the related books I just mentioned, this book’s scope follows two explicit constraints: • It does not cover Python language fundamentals • It is not intended as a language reference The former of these constraints reflects the fact that core language topics are the exclusive domain of Learning Python, and I encourage you to consult that book before tackling this one if you are completely new to the Python language, as its topics are assumed here. Some language techniques are shown by example in this book too, of course, and the larger examples here illustrate how core concepts come together into realistic programs. OOP, for example, is often best sampled in the context of the larger programs we’ll write here. Officially, though, this book assumes you already know enough Python fundamentals to understand its example code. Our focus here is mostly on libraries and tools; please see other resources if the basic code we’ll use in that role is unclear. The latter of the two constraints listed above reflects what has been a common misconception about this book over the years (indeed, this book might have been better titled Applying Python had we been more clairvoyant in 1995). I want to make this as clear as I can: this is not a reference book. It is a tutorial. Although you can hunt for some details using the index and table of contents, this book is not designed for that purpose. Instead, Python Pocket Reference provides the sort of quick reference to details that you’ll find useful once you start writing nontrivial code on your own. There are other reference-focused resources available, including other books and Python’s own reference manuals set. Here, the goal is a gradual tutorial that teaches you how to apply Python to common tasks but does not document minute details exhaustively. About This Fourth Edition If this is the first edition of this book you’ve seen, you’re probably less interested in recent changes, and you should feel free to skip ahead past this section. For readers of prior editions, though, this Fourth Edition of this book has changed in three important ways: • It’s been updated to cover Python 3.X (only). • It’s been slimmed down to sharpen its focus and make room for new topics. • It’s been updated for newly emerged topics and tools in the Python world. The first of these is probably the most significant—this edition employs the Python 3.X language, its version of the standard library, and the common practice of its users. To better explain how this and the other two changes take shape in this edition, though, I need to fill in a few more details. Preface | xxv Specific Changes in This Edition Because the prior versions of this book were widely read, here is a quick rundown of some of the most prominent specific changes in this edition: Its existing material was shortened to allow for new topics The prior edition of this book was also a 1600-page volume, which didn’t allow much room for covering new Python topics (Python 3.X’s Unicode orientation alone implies much new material). Luckily, recent changes in the Python world have allowed us to pare down some less critical existing material this time around, in order to free up room for new coverage. Depth was not sacrificed in the process, of course, and this is still just as substantial a book as before. In general, though, avoiding new growth was a primary goal of this update; many of the other specific changes and removals I'll mention below were made, in part, to help accommodate new topics. It covers 3.X (only) This book’s examples and narrative have been updated to reflect and use the 3.X version of Python. Python 2.X is no longer supported here, except where 3.X and 2.X Pythons overlap. Although the overlap is large enough to make this of use to 2.X readers too, this is now officially a 3.X-only text. This turns out to be a major factor behind the lack of growth in this edition. By restricting our scope to Python 3.X—the incompatible successor to the Python 2.X line, and considered to be Python’s future—we were able to avoid doubling the coverage size in places where the two Python lines differ. This version limit is especially important in a book like this that is largely about more advanced examples, which can be listed in only one version’s style. For readers who still straddle the 2.X and 3.X worlds, I’ll say more about Python 3.X changes later in this Preface. Probably the most significant 3.X-related change described there is the new Internationalization support in PyEdit and PyMailGUI; though 2.X had Unicode too, its new prominence in 3.X almost forces such systems to rethink their former ASCII-only ways. Inclusion of newly emerged libraries and tools Since the prior edition, a variety of new libraries and tools have either come online or risen in popularity, and they get new mention here. This includes new standard library tools such as subprocess (in Chapters 2 and 3) and multiprocessing (in Chapter 5), as well as new third-party web frameworks and ORM database toolkits. Most of these are not covered extensively (many popular third-party extensions are complex systems in their own right and are best covered by dedicated books), but they are at the least introduced in summary form here. For example, Python 3.1’s new tkinter.ttk Tk themed widget set shows up in Chapter 7 now, but only briefly; as a rule, this edition prefers to mention such extensions in passing, rather than attempting to show you code without adequate explanation. xxvi | Preface This Preface was tightened up I’ve removed all the instructions for using and running program examples. Instead, please consult the README file in the examples distribution for example usage details. Moreover, most of the original acknowledgments are gone here because they are redundant with those in Learning Python; since that book is now considered a prerequisite, duplication of material here is unwarranted. A description of book contents was also deleted; please see the table of contents for a preview of this book’s structure. The initial Python overview chapter is gone I’ve removed the prior edition’s “managerial summary” chapter which introduced Python’s strong points, prominent users, philosophies, and so on. Proselytizing does play an important role in a field that sometimes asks the “why” questions less often than it should. Indeed, if advocacy had not been part of the Python experience, we’d probably all be using Perl or shell languages today! However, this chapter has now grown completely redundant with a similar chapter in Learning Python. Since that book is a precursor to this one, I opted to not devote space to restating “Pythonista” propaganda here (fun as it may be). Instead, this book assumes you already know why Python is worth using, and we jump right into applying it here. The conclusion’s postscripts are gone This book’s conclusion comes from the first edition, and it is now 15 years old. Naturally, some of it reflects the Python mindset from that period more than that of today. For example, its focus on Python’s role in hybrid applications seemed more important in 1995 than in 2010; in today’s much larger Python world, most Python users never deal with linked-in C code at all. In prior editions, I added postscripts for each edition to elaborate on and update the ideas presented in the book’s conclusion. These postscripts are gone now, replaced by a short note at the start of the conclusion. I opted to keep the conclusion itself, though, because it’s still relevant to many readers and bears some historic value. Well, that, plus the jokes… The forewords are gone For reasons similar to those of the prior two points, the accumulated forewords from the prior three editions were also dropped this time around. You can read all about Python creator Guido van Rossum’s historical rationale for Python’s evolution in numerous places on the Web, if you are so inclined. If you are interested in how Python has changed technically over the years, see also the “What’s New” documents that are part of the Python standard manuals set (available at http:// www.python.org/doc, and installed alongside Python on Windows and other platforms). The C integration part has been reduced to just one chapter I’ve reduced the C extending and embedding part’s material to one shorter chapter at the end of the tools part, which briefly introduces the core concepts in this Preface | xxvii domain. Only a fraction of Python users must care about linking in C libraries today, and those who do already have the skills required to read the larger and more compete example of integration present in the source code of Python itself. There is still enough to hint at possibilities here, but vast amounts of C code have been cut, in deference to the better examples you’ll find in Python’s own code. The systems programming part was condensed and reworked The former two larger system examples chapters have been merged into one shorter one, with new or greatly rewritten examples. In fact, this part (Part II) was probably overhauled the most of any part in the book. It incorporates new tools such as subprocess and multiprocessing, introduces sockets earlier, and removes dated topics and examples still lingering from prior editions. Frankly, a few of the fileoriented examples here dated back to the 1990s, and were overdue for a general refresh. The initial chapter in this part was also split into two to make its material easier to read (shell context, including streams, gets its own chapter now), and a few large program listings here (including the auto-configuring launcher scripts) are now external suggested reading. Some larger examples were removed (but are available in the examples distribution) Along the same lines, two of the larger GUI examples in the prior edition, PyTree and PyForm, have been removed. Instead, their updated code is available in the book’s examples distribution package, as suggested supplemental reading. You’ll still find many larger examples covered and listed in this edition—including both GUI- and Web-based renderings of full-featured email clients, along with image viewers, calculators, clocks, Unicode-aware text editors, drawing programs, regression test scripts, and more. However, because the code of the examples removed doesn’t add much to what is already covered, and because they were already largely self-study examples anyhow, I’ve made them optional and external to the printed text in this edition. The advanced Internet topics chapter was replaced by brief summaries I’ve cut the advanced Internet topics chapter completely, leaving only simple summaries at the start of the Internet part (intentionally mirroring the GUI option summaries at the start of the GUI part). This includes prior coverage for tools such as the ZOPE web framework, COM, Windows active scripting and ASP, HTMLgen, Python Server Pages (PSP), Jython, and the now very dated Grail system. Some of these systems still receive honorable mention in the summaries, but none are now presented in any sort of detail. Summaries of new tools (including many of those listed in the following paragraph) were added to this set, but again, in brief fashion with no example code. Despite authors’ best attempts to foresee the future, the Web domain evolves faster than books like this can. For instance, Web frameworks like Django, Google’s App Engine, TurboGears, pylons, and web2py are now popular alternatives to ZOPE. Similarly, the .NET framework supersedes much of COM on Windows; IronPython now provides the same type of integration for .NET as Jython did first xxviii | Preface for Java; and active scripting has been eclipsed by AJAX and JavaScript-oriented frameworks on the client such as Flex, Silverlight, and pyjamas (generally known today as rich Internet applications, RIAs). Culture shift aside, the examples formerly presented in this category were by themselves also insufficient to either teach or do justice to the subject tools. Rather than including incomplete (and nearly useless) coverage of tools that are prone to both evolution and demise during this edition’s expected lifespan, I now provide only brief overviews of the current hot topics in the Web domain, and I encourage readers to search the Web for more details. More to the point, the goal of the book you’re reading is to impart the sort of in-depth knowledge of Internet and Web fundamentals that will allow you to use more advanced systems well, when you’re ready to take the leap. One exception here: the XML material of this prior chapter was spared and relocated in expanded form to the text processing chapter (where it probably belonged all along). In a related vein, the coverage of ZOPE’s ZODB object-oriented database was retained, although it was shortened radically to allow new coverage of ORMs such as SQLObject and SQLAlchemy (again, in overview form). Use of tools available for 3.X today At this writing, Python 3.X is still in its adoption phase, and some of the third-party tools that this book formerly employed in its examples are still available in Python 2.X form only. To work around this temporary flux, I’ve changed some code to use alternatives that already support 3.X today. The most notable of these is the SQL database section—this now uses the inprocess SQLite library, which is a standard part of Python and already in 3.X form, rather than the enterprise-level MySQL interface which is still at 2.X today. Luckily, the Python portable database API allows scripts to work largely the same on both, so this is a minor pragmatic sacrifice. Of special note, the PIL extension used to display JPEGs in the GUI part was ported to 3.1 just when it was needed for this update, thanks to Fredrik Lundh. It’s still not officially released in 3.X form as I submit the final draft of this book in July 2010, but it should be soon, and 3.X patches are provided in the book examples package as a temporary measure. Advanced core language topics are not covered here More advanced Python language tools such as descriptors, properties, decorators, metaclasses, and Unicode text processing basics are all part of the core Python language. Because of that, they are covered in the Fourth Edition of Learning Python, not here. For example, Unicode text and the changes it implies for files, filenames, sockets, and much more are discussed as encountered here, but the fundamentals of Unicode itself are not presented in complete depth. Some of the topics in this category are arguably application-level related too (or at least of interest to tool builders and API developers in general), but their coverage in Learning Preface | xxix Python allows us to avoid additional growth here. Please see that book for more on these subjects. Other random bits Naturally, there were additional smaller changes made along the way. For example, tkinter’s grid method is used instead of pack for layout of most input forms, because it yields a more consistent layout on platforms where label font sizes don’t match up with entry widget height (including on a Windows 7 netbook laptop, this edition’s development machine). There’s also new material scattered throughout, including a new exploration of redirecting streams to sockets in the Internet part; a new threaded and Unicode-aware “grep” dialog and process-wide change tests on exit in the PyEdit example; and other things you are probably better off uncovering along the way than reading further about in this Preface. I also finally replaced some remaining “#” comment blocks at the top of source files with docstrings (even, for consistency, in scripts not meant to be imported, though some “#” lines are retained in larger examples to offset the text); changed a few lingering “while 1” to “while True”; use += more often; and cleaned up a few other cases of now-dated coding patterns. Old habits may die hard, but such updates make the examples both more functional and more representative of common practice today. Although new topics were added, all told, four chapters were cut outright (the nontechnical introduction, one of the system example chapters, advanced Internet topics, and one integration chapter), some additional examples and material were trimmed (including PyForm and PyTree), and focus was deliberately restricted to Python 3.X and application fundamentals to conserve space. What’s Left, Then? The combined effect of all the changes just outlined is that this edition more concisely and sharply reflects its core focus—that of a tutorial introduction to ways to apply Python in common programming domains. Nevertheless, as you can tell from this book’s page count, it is still a substantial and in-depth book, designed to be a first step on your path to mastering realistic applications of Python. Contrary to recent trends (and at some risk of being branded a heretic), I firmly believe that the job of books like this one is to elevate their readers, not pander to them. Lowering the intellectual bar does a disservice both to readers and to the fields in which they hope to work. While that means you won’t find as many cartoons in this book as in some, this book also won’t insult you by emphasizing entertainment at the expense of technical depth. Instead, the goal of my books is to impart sophisticated concepts in a satisfying and substantive way and to equip you with the tools you’ll need in the real world of software development. xxx | Preface There are many types of learners, of course, and no one book can ever satisfy every possible audience. In fact, that’s why the original version of this book later became two, with language basics delegated to Learning Python. Moreover, one can make a case for a distinction between programmers, who must acquire deep software development skills, and scripters, who do not. For some, a rudimentary knowledge of programming may be enough to leverage a system or library that solves the problem at hand. That is, until their coding forays start encroaching on the realm of full-scale software engineering—a threshold that can inspire disappointment at worst, but a better appreciation of the challenging nature of this field at best. No matter which camp you’re from, it’s important to understand this book’s intent upfront. If you’re looking for a shortcut to proficiency that’s light on technical content, you probably won’t be happy with this book (or the software field in general). If your goal is to master programming Python well, though, and have some fun along the way, you’ll probably find this book to be an important piece of your learning experience. At the end of the day, learning to program well is much more demanding than implied by some contemporary media. If you’re willing to invest the focus and effort required, though, you’ll find that it’s also much more rewarding. This is especially true for those who equip themselves for the journey with a programmer-friendly tool like Python. While no book or class can turn you into a Python “Master of the Universe” by itself, this book’s goal is to help you get there, by shortening your start-up time and providing a solid foundation in Python’s most common application domains. Python 3.X Impacts on This Book As mentioned, this edition now covers Python 3.X only. Python 3.X is an incompatible version of the language. The 3.X core language itself is very similar to Python 2.X, but there are substantial changes in both the language and its many standard libraries. Although some readers with no prior background in 2.X may be able to bypass the differences, the changes had a big impact on the content of this edition. For the still very large existing Python 2.X user base, this section documents the most noteworthy changes in this category. If you’re interested in 2.X differences, I also suggest finding a copy of the Fourth Edition of the book Python Pocket Reference described earlier. That book gives both 2.X and 3.X versions of core language structures, built-in functions and exceptions, and many of the standard library modules and tools used in this book. Though not designed to be a reference or version translator per se, the Fourth Edition of Learning Python similarly covers both 2.X and 3.X, and as stated, is prerequisite material to this book. The goal of this 3.X-only Programming Python is not to abandon the current vast 2.X user base in favor of a still imaginary one for 3.X; it is to help readers with the migration, and avoid doubling the size of an already massive book. Preface | xxxi Specific 3.X Changes Luckily, many of the 2.X/3.X differences that impact this book’s presentation are trivial. For instance, the tkinter GUI toolkit, used extensively in this book, is shown under its 3.X tkinter name and package structure only; its 2.X Tkinter module incarnation is not described. This mostly boils down to different import statements, but only their Python 3 versions are given here. Similarly, to satisfy 3.X module naming conventions, 2.X’s anydbm, Queue, thread, StringIO.StringIO, and urllib.open become dbm, queue, _thread, io.StringIO, and urllib.request.urlopen, respectively, in both Python 3.X and this edition. Other tools are similarly renamed. On the other hand, 3.X implies broader idiomatic changes which are, of course, more radical. For example, Python 3.X’s new Unicode awareness has inspired fully Internationalized versions of the PyEdit text editor and the PyMailGUI email client examples in this edition (more on this in a moment). Furthermore: the replacement of os.popen2 with the subprocess module required new examples; the demise of os.path.walk in favor of os.walk allowed some examples to be trimmed; the new Unicode and binary dichotomy of files and strings impacted a host of additional existing examples and material; and new modules such as multiprocessing offer new options covered in this edition. Beyond such library changes, core language changes in Python 3 are also reflected in this book’s example code. For instance, changes to 2.X’s print, raw_input, keys, has_key, map, and apply all required changes here. In addition, 3.X’s new packagerelative import model impacted a few examples including mailtools and expression parsers, and its different flavor of division forced some minor math updates in canvasbased GUI examples such as PyClock, PyDraw, and PyPhoto. Of note here, I did not change all % string formatting expressions to use the new str.format, since both forms are supported in Python 3.1, and it now appears that they will be either indefinitely or forever. In fact, per a “grep” we’ll build and run in Chapter 11’s PyEdit example, it seems that this expression still appears over 3,000 times in Python 3.1’s own library code. Since I cannot predict Python evolution completely, see the first chapter for more on this if it ever requires updates in an unexpected future. Also because of the 3.X scope, this edition is unable to use some third-party packages that are still in 2.X form only, as described earlier. This includes the leading MySQL interface, ZODB, PyCrypto, and others; as also mentioned, PIL was ported to 3.1 for use in this book, but this required a special patch and an official 3.X release is still presently pending. Many of these may be available in 3.X form by the time you read these words, assuming the Python world can either break some of the current cross dependencies in 2.X packages or adopt new 3.X-only tools. xxxii | Preface Language Versus Library: Unicode As a book focused on applications instead of core language fundamentals, language changes are not always obtrusive here. Indeed, in retrospect the book Learning Python may have been affected by 3.X core language changes more than this book. In most cases here, more example changes were probably made in the name of clarity or functionality than in support of 3.X itself. On the other hand, Python 3.X does impact much code, and the impacts can be subtle at times. Readers with Python 2.X backgrounds will find that while 3.X core language changes are often simple to apply, updates required for changes in the 3.X standard library are sometimes more far reaching. Chief among these, Python 3.X’s Unicode strings have had broad ramifications. Let’s be honest: to people who have spent their lives in an ASCII world, the impacts of the 3.X Unicode model can be downright aggravating at times! As we’ll see in this book, it affects file content; file names; pipe descriptors; sockets; text in GUIs; Internet protocols such as FTP and email; CGI scripts; and even some persistence tools. For better or worse, once we reach the world of applications programming as covered in this book, Unicode is no longer an optional topic for many or most Python 3.X programmers. Of course, Unicode arguably never should have been entirely optional for many programmers in the first place. Indeed, we’ll find that things that may have appeared to work in 2.X never really did—treating text as raw byte strings can mask issues such as comparison results across encodings (see the grep utility of Chapter 11’s PyEdit for a prime example of code that should fail in the face of Unicode mismatches). Python 3.X elevates such issues to potentially every programmer’s panorama. Still, porting nontrivial code to 3.X is not at all an insurmountable task. Moreover, many readers of this edition have the luxury of approaching Python 3.X as their first Python and need not deal with existing 2.X code. If this is your case, you’ll find Python 3.X to be a robust and widely applicable scripting and programming language, which addresses head-on many issues that once lurked in the shadows in 2.X. Python 3.1 Limitations: Email, CGI There’s one exception that I should call out here because of its impact on major book examples. In order to make its code relevant to the widest possible audience, this book’s major examples are related to Internet email and have much new support in this edition for Internationalization and Unicode in this domain. Chapter 14’s PyMailGUI and Chapter 16’s PyMailCGI, and all the prior examples they reuse, fall into this category. This includes the PyEdit text editor—now Unicode-aware for files, display, and greps. On this front, there is both proverbial good news and bad. The good news is that in the end, we will be able to develop the feature-rich and fully Internationalized PyMailGUI email client in this book, using the email package as it currently exists. This will include support for arbitrary encodings in both text content and message headers, for Preface | xxxiii both viewing and composing messages. The less happy news is that this will come at some cost in workaround complexity in Python 3.1. Unfortunately, as we’ll learn in Chapter 13, the email package in Python 3.1 has a number of issues related to str/bytes combinations in Python 3.X. For example, there’s no simple way to guess the encoding needed to convert mail bytes returned by the poplib module to the str expected by the email parser. Moreover, the email package is currently broken altogether for some types of messages, and it has uneven or typespecific support for some others. This situation appears to be temporary. Some of the issues encountered in this book are already scheduled to be repaired (in fact, one such fix in 3.2 required a last-minute patch to one of this book’s 3.1 workarounds in Chapter 13). Furthermore, a new version of email is being developed to accommodate the 3.X Unicode/bytes dichotomy more accurately, but it won’t materialize until long after this book is published, and it might be backward-incompatible with the current package’s API, much like Python 3.X itself. Because of that, this book both codes workarounds and makes some assumption along the way, but please watch its website (described ahead) for required updates in future Pythons. One upside here is that the dilemmas posed neatly reflect those common in realistic programming—an underlying theme of this text. These issues in the email package are also inherited by the cgi module for CGI file uploads, which are in large measure broken in 3.1. CGI scripts are a basic technique eclipsed by many web frameworks today, but they still serve as an entry-level way to learn Web fundamentals and are still at the heart of many larger toolkits. A future fix seems likely for this 3.1 flaw as well, but we have to make do with nonbinary CGI file uploads for this edition in Chapters 15 and 16, and limited email attachments in PyMailCGI. This seems less than ideal nearly two years after 3.0’s release, but such is life in the dynamic worlds of both software development at large and books that aim to lead the curve instead of following it. Using Book Examples Because this book’s examples form much of its content, I want to say a few words about them up front. Where to Look for Examples and Updates As before, examples, updates, corrections, and supplements for this book will be maintained at the author’s website, which lives officially at the following URL: http://www.rmi.net/~lutz/about-pp4e.html This page at my book support website will contain links to all supplemental information related to this version of the book. Because I don’t own that domain name, though, if xxxiv | Preface that link ceases to be during this book’s shelf life, try the following alternative site as a fallback option: http://learning-python.com/books/about-pp4e.html (alternative location) If neither of those links work, try a general web search (which, of course, is what most readers will probably try first anyhow). Wherever it may live, this website (as well as O’Reilly’s, described in the next section) is where you can fetch the book examples distribution package—an archive file containing all of the book’s examples, as well as some extras that are mentioned but not listed in the book itself. To work along without having to type the examples manually, download the package, unpack it, and consult its README.txt file for usage details. I’ll describe how example labels and system prompts in this book imply file locations in the package when we use our first script in the first chapter. As for the first three editions, I will also be maintaining an informal “blog” on this website that describes Python changes over time and provides general book-related notes and updates that you should consider a supplemental appendix to this text. O’Reilly’s website for this book, described later in this Preface, also has an errata report system, and you can report issues at either my site or O’Reilly’s. I tend to keep my book websites more up to date, but it’s not impossible that O’Reilly’s errata page may supersede mine for this edition. In any event, you should consider the union of these two lists to be the official word on book corrections and updates. Example Portability The examples in this book were all developed, tested, and run under Windows 7, and Python 3.1. The book’s major examples were all tested and ran successfully on the upcoming Python 3.2, too (its alpha 3 release), just before the book went to the printer, so most or all of this book applies to Python 3.2 as well. In addition, the C code of Chapter 20 and a handful of parallel programming examples were run under Cygwin on Windows to emulate a Unix environment. Although Python and its libraries are generally platform neutral, some of this book’s code may require minor changes to run on other platforms, such as Mac OS X, Linux, and other Unix variants. The tkinter GUI examples, as well as some systems programming scripts, may be especially susceptible to platform differences. Some portability issues are pointed out along the way, but others may not be explicitly noted. Since I had neither time nor budget to test on and accommodate all possible machines that readers might use over the lifespan of this book, updates for platform-specific behaviors will have to fall into the suggested exercises category. If you find a platform dependency and wish to submit a patch for it, though, please see the updates site listed earlier; I’ll be happy to post any platform patches from readers there. Preface | xxxv Demo Launchers The book examples package described earlier also includes portable example demo launcher scripts named PyDemos and PyGadgets, which provide a quick look at some of this book’s major GUI- and Web-based examples. These scripts and their launchers, located at the top of the examples tree, can be run to self-configure program and module search paths, and so can generally be run immediately on compatible platforms, including Windows. See the package’s README files as well as the overviews near the end of Chapters 6 and 10 for more on these scripts. Code Reuse Policies We now interrupt this Preface for a word from the legal department. This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Programming Python, Fourth Edition, by Mark Lutz (O’Reilly). Copyright 2011 Mark Lutz, 978-0-596-15810-1.” Contacting O’Reilly I described my own examples and updates sites in the prior section. In addition to that advice, you can also address comments and questions about this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States and Canada) 707-827-7000 (international/local) 707-829-0104 (fax) As mentioned, O’Reilly maintains a web page for this book, which lists errata, examples, and any additional information. You can access this page at: http://oreilly.com/catalog/9780596158101 To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com xxxvi | Preface For more information about books, conferences, software, Resource Centers, and the O’Reilly Network, see the O’Reilly website at: http://www.oreilly.com Conventions Used in This Book The following font conventions are used in this book: Italic Used for file and directory names, to emphasize new terms when first introduced, and for some comments within code sections Constant width Used for code listings and to designate modules, methods, options, classes, functions, statements, programs, objects, and HTML tags Constant width bold Used in code sections to show user input Constant width italic Used to mark replaceables This icon designates a note related to the nearby text. This icon designates a warning related to the nearby text. Acknowledgments I acknowledged numerous people in the preface of Learning Python, Fourth Edition, less than a year ago; because that book is a precursor to this one, and because the set is largely the same, I won’t repeat the list in its entirety here. In short, though, I’m grateful to: • O’Reilly, for promoting Python, and publishing “meaty” books in the Open Source domain • The Python community, which has occupied sizeable portions of my world since 1992 • The thousands of students who attended the 250 Python classes I’ve taught since 1997 Preface | xxxvii • The hundreds of thousands who read the 12 editions of the three Python books I’ve written since 1995 • Monty Python, Python’s namesake, for so many great bits to draw from (more in the next chapter) Although writing is ultimately a solitary task, the ideas that spring forth owe much to the input of many. I’m thankful for all the feedback I’ve been fortunate to receive over the last 18 years, both from classes and from readers. Students really are the best teachers of teachers. On the (overly) personal front, I’d like to thank my brothers and sister for old days, as well as my children, Michael, Samantha, and Roxanne, for bragging rights. And I’m especially thankful for my wife, Vera, who somehow managed to append very good things to this otherwise immutable object. —Mark Lutz, July 2010 xxxviii | Preface So, What’s Python? As discussed, this book won’t devote much space to Python fundamentals, and we’ll defer an abstract discussion of Python roles until the Conclusion, after you’ve had a chance to see it in action firsthand. If you are looking for a concise definition of this book’s topic, though, try this: Python is a general-purpose, open source computer programming language. It is optimized for software quality, developer productivity, program portability, and component integration. Python is used by at least hundreds of thousands of developers around the world in areas such as Internet scripting, systems programming, user interfaces, product customization, numeric programming, and more. It is generally considered to be among the top four or five most widely-used programming languages in the world today. As a popular language focused on shrinking development time, Python is deployed in a wide variety of products and roles. Counted among its current user base are Google, YouTube, Industrial Light & Magic, ESRI, the BitTorrent file sharing system, NASA’s Jet Propulsion Lab, the game Eve Online, and the National Weather Service. Python’s application domains range from system administration, website development, cell phone scripting, and education to hardware testing, investment analysis, computer games, and spacecraft control. Among other things, Python sports a remarkably simple, readable, and maintainable syntax; integration with external components coded in other languages; a multiparadigm design, with OOP, functional, and modular structures; and a vast collection of precoded interfaces and utilities. Its tool set makes it a flexible and agile language, ideal for both quick tactical tasks as well as longer-range strategic application development efforts. Although it is a general-purpose language, Python is often called a scripting language because it makes it easy to utilize and direct other software components. Perhaps Python’s best asset, though, is simply that it makes software development more rapid and enjoyable. There is a class of people for whom programming is an end in itself. They enjoy the challenge. They write software for the pure pleasure of doing so and often view commercial or career reward as secondary consequence. This is the class that largely invented the Internet, open source, and Python. This is also the class that has historically been a primary audience for this book. As they’ve often relayed, with a tool like Python, programming can be just plain fun. To truly understand how, read on; though something of a side effect, much of this book serves as a demonstration of Python’s ideals in action in real-world code. As we’ll see, especially when combined with toolkits for GUIs, websites, systems programming, and so on, Python serves as enabling technology. Preface | xxxix PART I The Beginning This part of the book gets things started by taking us on a quick tour that reviews Python fundamental prerequisites and introduces some of the most common ways it is applied. Chapter 1 This chapter kicks things off by using a simple example—recording information about people—to briefly introduce some of the major Python application domains we’ll be studying in this book. We’ll migrate the same example through multiple steps. Along the way, we’ll meet databases, GUIs, websites, and more. This is something of a demo chapter, designed to pique your interest. We won’t learn the full story here, but we’ll have a chance to see Python in action before digging into the details. This chapter also serves as a review of some core language ideas you should be familiar with before starting this book, such as data representation and object-oriented programming (OOP). The point of this part of the book is not to give you an in-depth look at Python, but just to let you sample its application and to provide you with a quick look at some of Python’s broader goals and purposes. CHAPTER 1 A Sneak Preview “Programming Python: The Short Story” If you are like most people, when you pick up a book as large as this one, you’d like to know a little about what you’re going to be learning before you roll up your sleeves. That’s what this chapter is for—it provides a demonstration of some of the kinds of things you can do with Python, before getting into the details. You won’t learn the full story here, and if you’re looking for complete explanations of the tools and techniques applied in this chapter, you’ll have to read on to later parts of the book. The point here is just to whet your appetite, review a few Python basics, and preview some of the topics to come. To do this, I’ll pick a fairly simple application task—constructing a database of records—and migrate it through multiple steps: interactive coding, command-line tools, console interfaces, GUIs, and simple web-based interfaces. Along the way, we’ll also peek at concepts such as data representation, object persistence, and objectoriented programming (OOP); explore some alternatives that we’ll revisit later in the book; and review some core Python ideas that you should be aware of before reading this book. Ultimately, we’ll wind up with a database of Python class instances, which can be browsed and changed from a variety of interfaces. I’ll cover additional topics in this book, of course, but the techniques you will see here are representative of some of the domains we’ll explore later. And again, if you don’t completely understand the programs in this chapter, don’t worry because you shouldn’t—not yet anyway. This is just a Python demo. We’ll fill in the rest of the details soon enough. For now, let’s start off with a bit of fun. 3 Readers of the Fourth Edition of Learning Python might recognize some aspects of the running example used in this chapter—the characters here are similar in spirit to those in the OOP tutorial chapter in that book, and the later class-based examples here are essentially a variation on a theme. Despite some redundancy, I’m revisiting the example here for three reasons: it serves its purpose as a review of language fundamentals; some readers of this book haven’t read Learning Python; and the example receives expanded treatment here, with the addition of GUI and Web interfaces. That is, this chapter picks up where Learning Python left off, pushing this core language example into the realm of realistic applications—which, in a nutshell, reflects the purpose of this book. The Task Imagine, if you will, that you need to keep track of information about people for some reason. Maybe you want to store an address book on your computer, or perhaps you need to keep track of employees in a small business. For whatever reason, you want to write a program that keeps track of details about these people. In other words, you want to keep records in a database—to permanently store lists of people’s attributes on your computer. Naturally, there are off-the-shelf programs for managing databases like these. By writing a program for this task yourself, however, you’ll have complete control over its operation. You can add code for special cases and behaviors that precoded software may not have anticipated. You won’t have to install and learn to use yet another database product. And you won’t be at the mercy of a software vendor to fix bugs or add new features. You decide to write a Python program to manage your people. Step 1: Representing Records If we’re going to store records in a database, the first step is probably deciding what those records will look like. There are a variety of ways to represent information about people in the Python language. Built-in object types such as lists and dictionaries are often sufficient, especially if we don’t initially care about processing the data we store. Using Lists Lists, for example, can collect attributes about people in a positionally ordered way. Start up your Python interactive interpreter and type the following two statements: >>> bob = ['Bob Smith', 42, 30000, 'software'] >>> sue = ['Sue Jones', 45, 40000, 'hardware'] 4 | Chapter 1: A Sneak Preview We’ve just made two records, albeit simple ones, to represent two people, Bob and Sue (my apologies if you really are Bob or Sue, generically or otherwise*). Each record is a list of four properties: name, age, pay, and job fields. To access these fields, we simply index by position; the result is in parentheses here because it is a tuple of two results: >>> bob[0], sue[2] ('Bob Smith', 40000) # fetch name, pay Processing records is easy with this representation; we just use list operations. For example, we can extract a last name by splitting the name field on blanks and grabbing the last part, and we can give someone a raise by changing their list in-place: >>> bob[0].split()[-1] # what's bob's last name? 'Smith' >>> sue[2] *= 1.25 # give sue a 25% raise >>> sue ['Sue Jones', 45, 50000.0, 'hardware'] The last-name expression here proceeds from left to right: we fetch Bob’s name, split it into a list of substrings around spaces, and index his last name (run it one step at a time to see how). Start-up pointers Since this is the first code in this book, here are some quick pragmatic pointers for reference: • This code may be typed in the IDLE GUI; after typing python at a shell prompt (or the full directory path to it if it’s not on your system path); and so on. • The >>> characters are Python’s prompt (not code you type yourself). • The informational lines that Python prints when this prompt starts up are usually omitted in this book to save space. • I’m running all of this book’s code under Python 3.1; results in any 3.X release should be similar (barring unforeseeable Python changes, of course). • Apart from some system and C integration code, most of this book’s examples are run under Windows 7, though thanks to Python portability, it generally doesn’t matter unless stated otherwise. If you’ve never run Python code this way before, see an introductory resource such as O’Reilly’s Learning Python for help with getting started. I’ll also have a few words to say about running code saved in script files later in this chapter. * No, I’m serious. In the Python classes I teach, I had for many years regularly used the name “Bob Smith,” age 40.5, and jobs “developer” and “manager” as a supposedly fictitious database record—until a class in Chicago, where I met a student named Bob Smith, who was 40.5 and was a developer and manager. The world is stranger than it seems. Step 1: Representing Records | 5 A database list Of course, what we’ve really coded so far is just two variables, not a database; to collect Bob and Sue into a unit, we might simply stuff them into another list: >>> people = [bob, sue] >>> for person in people: print(person) # reference in list of lists ['Bob Smith', 42, 30000, 'software'] ['Sue Jones', 45, 50000.0, 'hardware'] Now the people list represents our database. We can fetch specific records by their relative positions and process them one at a time, in loops: >>> people[1][0] 'Sue Jones' >>> for person in people: print(person[0].split()[-1]) person[2] *= 1.20 # print last names # give each a 20% raise Smith Jones >>> for person in people: print(person[2]) # check new pay 36000.0 60000.0 Now that we have a list, we can also collect values from records using some of Python’s more powerful iteration tools, such as list comprehensions, maps, and generator expressions: >>> pays = [person[2] for person in people] >>> pays [36000.0, 60000.0] # collect all pay >>> pays = map((lambda x: x[2]), people) >>> list(pays) [36000.0, 60000.0] # ditto (map is a generator in 3.X) >>> sum(person[2] for person in people) 96000.0 # generator expression, sum built-in To add a record to the database, the usual list operations, such as append and extend, will suffice: >>> people.append(['Tom', 50, 0, None]) >>> len(people) 3 >>> people[-1][0] 'Tom' Lists work for our people database, and they might be sufficient for some programs, but they suffer from a few major flaws. For one thing, Bob and Sue, at this point, are 6 | Chapter 1: A Sneak Preview just fleeting objects in memory that will disappear once we exit Python. For another, every time we want to extract a last name or give a raise, we’ll have to repeat the kinds of code we just typed; that could become a problem if we ever change the way those operations work—we may have to update many places in our code. We’ll address these issues in a few moments. Field labels Perhaps more fundamentally, accessing fields by position in a list requires us to memorize what each position means: if you see a bit of code indexing a record on magic position 2, how can you tell it is extracting a pay? In terms of understanding the code, it might be better to associate a field name with a field value. We might try to associate names with relative positions by using the Python range builtin function, which generates successive integers when used in iteration contexts (such as the sequence assignment used initially here): >>> NAME, AGE, PAY = range(3) >>> bob = ['Bob Smith', 42, 10000] >>> bob[NAME] 'Bob Smith' >>> PAY, bob[PAY] (2, 10000) # 0, 1, and 2 This addresses readability: the three uppercase variables essentially become field names. This makes our code dependent on the field position assignments, though— we have to remember to update the range assignments whenever we change record structure. Because they are not directly associated, the names and records may become out of sync over time and require a maintenance step. Moreover, because the field names are independent variables, there is no direct mapping from a record list back to its field’s names. A raw record list, for instance, provides no way to label its values with field names in a formatted display. In the preceding record, without additional code, there is no path from value 42 to label AGE: bob.index(42) gives 1, the value of AGE, but not the name AGE itself. We might also try this by using lists of tuples, where the tuples record both a field name and a value; better yet, a list of lists would allow for updates (tuples are immutable). Here’s what that idea translates to, with slightly simpler records: >>> bob = [['name', 'Bob Smith'], ['age', 42], ['pay', 10000]] >>> sue = [['name', 'Sue Jones'], ['age', 45], ['pay', 20000]] >>> people = [bob, sue] This really doesn’t fix the problem, though, because we still have to index by position in order to fetch fields: >>> for person in people: print(person[0][1], person[2][1]) # name, pay Step 1: Representing Records | 7 Bob Smith 10000 Sue Jones 20000 >>> [person[0][1] for person in people] ['Bob Smith', 'Sue Jones'] >>> for person in people: print(person[0][1].split()[-1]) person[2][1] *= 1.10 # collect names # get last names # give a 10% raise Smith Jones >>> for person in people: print(person[2]) ['pay', 11000.0] ['pay', 22000.0] All we’ve really done here is add an extra level of positional indexing. To do better, we might inspect field names in loops to find the one we want (the loop uses tuple assignment here to unpack the name/value pairs): >>> for person in people: for (name, value) in person: if name == 'name': print(value) # find a specific field Bob Smith Sue Jones Better yet, we can code a fetcher function to do the job for us: >>> def field(record, label): for (fname, fvalue) in record: if fname == label: return fvalue # find any field by name >>> field(bob, 'name') 'Bob Smith' >>> field(sue, 'pay') 22000.0 >>> for rec in people: print(field(rec, 'age')) # print all ages 42 45 If we proceed down this path, we’ll eventually wind up with a set of record interface functions that generically map field names to field data. If you’ve done any Python coding in the past, though, you probably already know that there is an easier way to code this sort of association, and you can probably guess where we’re headed in the next section. 8 | Chapter 1: A Sneak Preview Using Dictionaries The list-based record representations in the prior section work, though not without some cost in terms of performance required to search for field names (assuming you need to care about milliseconds and such). But if you already know some Python, you also know that there are more efficient and convenient ways to associate property names and values. The built-in dictionary object is a natural: >>> bob = {'name': 'Bob Smith', 'age': 42, 'pay': 30000, 'job': 'dev'} >>> sue = {'name': 'Sue Jones', 'age': 45, 'pay': 40000, 'job': 'hdw'} Now, Bob and Sue are objects that map field names to values automatically, and they make our code more understandable and meaningful. We don’t have to remember what a numeric offset means, and we let Python search for the value associated with a field’s name with its efficient dictionary indexing: >>> bob['name'], sue['pay'] ('Bob Smith', 40000) # not bob[0], sue[2] >>> bob['name'].split()[-1] 'Smith' >>> sue['pay'] *= 1.10 >>> sue['pay'] 44000.0 Because fields are accessed mnemonically now, they are more meaningful to those who read your code (including you). Other ways to make dictionaries Dictionaries turn out to be so useful in Python programming that there are even more convenient ways to code them than the traditional literal syntax shown earlier—e.g., with keyword arguments and the type constructor, as long as the keys are all strings: >>> bob >>> sue >>> bob {'pay': >>> sue {'pay': = dict(name='Bob Smith', age=42, pay=30000, job='dev') = dict(name='Sue Jones', age=45, pay=40000, job='hdw') 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} by filling out a dictionary one field at a time (recall that dictionary keys are pseudorandomly ordered): >>> sue = {} >>> sue['name'] = 'Sue Jones' >>> sue['age'] = 45 >>> sue['pay'] = 40000 >>> sue['job'] = 'hdw' >>> sue {'job': 'hdw', 'pay': 40000, 'age': 45, 'name': 'Sue Jones'} Step 1: Representing Records | 9 and by zipping together name/value lists: >>> names = ['name', 'age', 'pay', 'job'] >>> values = ['Sue Jones', 45, 40000, 'hdw'] >>> list(zip(names, values)) [('name', 'Sue Jones'), ('age', 45), ('pay', 40000), ('job', 'hdw')] >>> sue = dict(zip(names, values)) >>> sue {'job': 'hdw', 'pay': 40000, 'age': 45, 'name': 'Sue Jones'} We can even make dictionaries from a sequence of key values and an optional starting value for all the keys (handy to initialize an empty dictionary): >>> fields = ('name', 'age', 'job', 'pay') >>> record = dict.fromkeys(fields, '?') >>> record {'job': '?', 'pay': '?', 'age': '?', 'name': '?'} Lists of dictionaries Regardless of how we code them, we still need to collect our dictionary-based records into a database; a list does the trick again, as long as we don’t require access by key at the top level: >>> bob {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} >>> sue {'job': 'hdw', 'pay': 40000, 'age': 45, 'name': 'Sue Jones'} >>> people = [bob, sue] >>> for person in people: print(person['name'], person['pay'], sep=', ') # reference in a list # all name, pay Bob Smith, 30000 Sue Jones, 40000 >>> for person in people: if person['name'] == 'Sue Jones': print(person['pay']) # fetch sue's pay 40000 Iteration tools work just as well here, but we use keys rather than obscure positions (in database terms, the list comprehension and map in the following code project the database on the “name” field column): >>> names = [person['name'] for person in people] >>> names ['Bob Smith', 'Sue Jones'] # collect names >>> list(map((lambda x: x['name']), people)) ['Bob Smith', 'Sue Jones'] # ditto, generate >>> sum(person['pay'] for person in people) 70000 # sum all pay 10 | Chapter 1: A Sneak Preview Interestingly, tools such as list comprehensions and on-demand generator expressions can even approach the utility of SQL queries here, albeit operating on in-memory objects: >>> [rec['name'] for rec in people if rec['age'] >= 45] ['Sue Jones'] # SQL-ish query >>> [(rec['age'] ** 2 if rec['age'] >= 45 else rec['age']) for rec in people] [42, 2025] >>> G = (rec['name'] for rec in people if rec['age'] >= 45) >>> next(G) 'Sue Jones' >>> G = ((rec['age'] ** 2 if rec['age'] >= 45 else rec['age']) for rec in people) >>> G.__next__() 42 And because dictionaries are normal Python objects, these records can also be accessed and updated with normal Python syntax: >>> for person in people: print(person['name'].split()[-1]) person['pay'] *= 1.10 # last name # a 10% raise Smith Jones >>> for person in people: print(person['pay']) 33000.0 44000.0 Nested structures Incidentally, we could avoid the last-name extraction code in the prior examples by further structuring our records. Because all of Python’s compound datatypes can be nested inside each other and as deeply as we like, we can build up fairly complex information structures easily—simply type the object’s syntax, and Python does all the work of building the components, linking memory structures, and later reclaiming their space. This is one of the great advantages of a scripting language such as Python. The following, for instance, represents a more structured record by nesting a dictionary, list, and tuple inside another dictionary: >>> bob2 = {'name': 'age': 'job': 'pay': {'first': 'Bob', 'last': 'Smith'}, 42, ['software', 'writing'], (40000, 50000)} Step 1: Representing Records | 11 Because this record contains nested structures, we simply index twice to go two levels deep: >>> bob2['name'] {'last': 'Smith', 'first': 'Bob'} >>> bob2['name']['last'] 'Smith' >>> bob2['pay'][1] 50000 # bob's full name # bob's last name # bob's upper pay The name field is another dictionary here, so instead of splitting up a string, we simply index to fetch the last name. Moreover, people can have many jobs, as well as minimum and maximum pay limits. In fact, Python becomes a sort of query language in such cases—we can fetch or change nested data with the usual object operations: >>> for job in bob2['job']: print(job) software writing # all of bob's jobs >> bob2['job'][-1] # bob's last job 'writing' >>> bob2['job'].append('janitor') # bob gets a new job >>> bob2 {'job': ['software', 'writing', 'janitor'], 'pay': (40000, 50000), 'age': 42, 'name': {'last': 'Smith', 'first': 'Bob'}} It’s OK to grow the nested list with append, because it is really an independent object. Such nesting can come in handy for more sophisticated applications; to keep ours simple, we’ll stick to the original flat record structure. Dictionaries of dictionaries One last twist on our people database: we can get a little more mileage out of dictionaries here by using one to represent the database itself. That is, we can use a dictionary of dictionaries—the outer dictionary is the database, and the nested dictionaries are the records within it. Rather than a simple list of records, a dictionary-based database allows us to store and retrieve records by symbolic key: >>> bob = dict(name='Bob Smith', age=42, pay=30000, job='dev') >>> sue = dict(name='Sue Jones', age=45, pay=40000, job='hdw') >>> bob {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} >>> db = {} >>> db['bob'] = bob >>> db['sue'] = sue >>> >>> db['bob']['name'] 'Bob Smith' >>> db['sue']['pay'] = 50000 >>> db['sue']['pay'] 50000 12 | Chapter 1: A Sneak Preview # reference in a dict of dicts # fetch bob's name # change sue's pay # fetch sue's pay Notice how this structure allows us to access a record directly instead of searching for it in a loop—we get to Bob’s name immediately by indexing on key bob. This really is a dictionary of dictionaries, though you won’t see all the gory details unless you display the database all at once (the Python pprint pretty-printer module can help with legibility here): >>> db {'bob': {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'}, 'sue': {'pay': 50000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'}} >>> import pprint >>> pprint.pprint(db) {'bob': {'age': 42, 'job': 'dev', 'name': 'Bob Smith', 'pay': 30000}, 'sue': {'age': 45, 'job': 'hdw', 'name': 'Sue Jones', 'pay': 50000}} If we still need to step through the database one record at a time, we can now rely on dictionary iterators. In recent Python releases, a dictionary iterator produces one key in a for loop each time through (for compatibility with earlier releases, we can also call the db.keys method explicitly in the for loop rather than saying just db, but since Python 3’s keys result is a generator, the effect is roughly the same): >>> for key in db: print(key, '=>', db[key]['name']) bob => Bob Smith sue => Sue Jones >>> for key in db: print(key, '=>', db[key]['pay']) bob => 30000 sue => 50000 To visit all records, either index by key as you go: >>> for key in db: print(db[key]['name'].split()[-1]) db[key]['pay'] *= 1.10 Smith Jones or step through the dictionary’s values to access records directly: >>> for record in db.values(): print(record['pay']) 33000.0 55000.0 >>> x = [db[key]['name'] for key in db] >>> x ['Bob Smith', 'Sue Jones'] >>> x = [rec['name'] for rec in db.values()] Step 1: Representing Records | 13 >>> x ['Bob Smith', 'Sue Jones'] And to add a new record, simply assign it to a new key; this is just a dictionary, after all: >>> db['tom'] = dict(name='Tom', age=50, job=None, pay=0) >>> >>> db['tom'] {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} >>> db['tom']['name'] 'Tom' >>> list(db.keys()) ['bob', 'sue', 'tom'] >>> len(db) 3 >>> [rec['age'] for rec in db.values()] [42, 45, 50] >>> [rec['name'] for rec in db.values() if rec['age'] >= 45] ['Sue Jones', 'Tom'] # SQL-ish query Although our database is still a transient object in memory, it turns out that this dictionary-of-dictionaries format corresponds exactly to a system that saves objects permanently—the shelve (yes, this should probably be shelf, grammatically speaking, but the Python module name and term is shelve). To learn how, let’s move on to the next section. Step 2: Storing Records Persistently So far, we’ve settled on a dictionary-based representation for our database of records, and we’ve reviewed some Python data structure concepts along the way. As mentioned, though, the objects we’ve seen so far are temporary—they live in memory and they go away as soon as we exit Python or the Python program that created them. To make our people persistent, they need to be stored in a file of some sort. Using Formatted Files One way to keep our data around between program runs is to write all the data out to a simple text file, in a formatted way. Provided the saving and loading tools agree on the format selected, we’re free to use any custom scheme we like. Test data script So that we don’t have to keep working interactively, let’s first write a script that initializes the data we are going to store (if you’ve done any Python work in the past, you know that the interactive prompt tends to become tedious once you leave the realm of simple one-liners). Example 1-1 creates the sort of records and database dictionary we’ve been working with so far, but because it is a module, we can import it repeatedly without having to retype the code each time. In a sense, this module is a database itself, but its program code format doesn’t support automatic or end-user updates as is. 14 | Chapter 1: A Sneak Preview Example 1-1. PP4E\Preview\initdata.py # initialize data to be stored in files, pickles, shelves # records bob = {'name': 'Bob Smith', 'age': 42, 'pay': 30000, 'job': 'dev'} sue = {'name': 'Sue Jones', 'age': 45, 'pay': 40000, 'job': 'hdw'} tom = {'name': 'Tom', 'age': 50, 'pay': 0, 'job': None} # database db = {} db['bob'] = bob db['sue'] = sue db['tom'] = tom if __name__ == '__main__': # when run as a script for key in db: print(key, '=>\n ', db[key]) As usual, the __name__ test at the bottom of Example 1-1 is true only when this file is run, not when it is imported. When run as a top-level script (e.g., from a command line, via an icon click, or within the IDLE GUI), the file’s self-test code under this test dumps the database’s contents to the standard output stream (remember, that’s what print function-call statements do by default). Here is the script in action being run from a system command line on Windows. Type the following command in a Command Prompt window after a cd to the directory where the file is stored, and use a similar console window on other types of computers: ...\PP4E\Preview> python initdata.py bob => {'job': 'dev', 'pay': 30000, 'age': 42, 'name': 'Bob Smith'} sue => {'job': 'hdw', 'pay': 40000, 'age': 45, 'name': 'Sue Jones'} tom => {'job': None, 'pay': 0, 'age': 50, 'name': 'Tom'} File name conventions Since this is our first source file (a.k.a. “script”), here are three usage notes for this book’s examples: • The text ...\PP4E\Preview> in the first line of the preceding example listing stands for your operating system’s prompt, which can vary per platform; you type just the text that follows this prompt (python initdata.py). • Like all examples in this book, the system prompt also gives the directory in the downloadable book examples package where this command should be run. When running this script using a command-line in a system shell, make sure the shell’s current working directory is PP4E\Preview. This can matter for examples that use files in the working directory. Step 2: Storing Records Persistently | 15 • Similarly, the label that precedes every example file’s code listing tells you where the source file resides in the examples package. Per the Example 1-1 listing label shown earlier, this script’s full filename is PP4E\Preview\initdata.py in the examples tree. We’ll use these conventions throughout the book; see the Preface for more on getting the examples if you wish to work along. I occasionally give more of the directory path in system prompts when it’s useful to provide the extra execution context, especially in the system part of the book (e.g., a “C:\” prefix from Windows or more directory names). Script start-up pointers I gave pointers for using the interactive prompt earlier. Now that we’ve started running script files, here are also a few quick startup pointers for using Python scripts in general: • On some platforms, you may need to type the full directory path to the Python program on your machine; if Python isn’t on your system path setting on Windows, for example, replace python in the command with C:\Python31\python (this assumes you’re using Python 3.1). • On most Windows systems you also don’t need to type python on the command line at all; just type the file’s name to run it, since Python is registered to open “.py” script files. • You can also run this file inside Python’s standard IDLE GUI (open the file and use the Run menu in the text edit window), and in similar ways from any of the available third-party Python IDEs (e.g., Komodo, Eclipse, NetBeans, and the Wing IDE). • If you click the program’s file icon to launch it on Windows, be sure to add an input() call to the bottom of the script to keep the output window up. On other systems, icon clicks may require a #! line at the top and executable permission via a chmod command. I’ll assume here that you’re able to run Python code one way or another. Again, if you’re stuck, see other books such as Learning Python for the full story on launching Python programs. Data format script Now, all we have to do is store all of this in-memory data in a file. There are a variety of ways to accomplish this; one of the most basic is to write one piece of data at a time, with separators between each that we can use when reloading to break the data apart. Example 1-2 shows one way to code this idea. 16 | Chapter 1: A Sneak Preview Example 1-2. PP4E\Preview\make_db_file.py """ Save in-memory database object to a file with custom formatting; assume 'endrec.', 'enddb.', and '=>' are not used in the data; assume db is dict of dict; warning: eval can be dangerous - it runs strings as code; could also eval() record dict all at once; could also dbfile.write(key + '\n') vs print(key, file=dbfile); """ dbfilename = 'people-file' ENDDB = 'enddb.' ENDREC = 'endrec.' RECSEP = '=>' def storeDbase(db, dbfilename=dbfilename): "formatted dump of database to flat file" dbfile = open(dbfilename, 'w') for key in db: print(key, file=dbfile) for (name, value) in db[key].items(): print(name + RECSEP + repr(value), file=dbfile) print(ENDREC, file=dbfile) print(ENDDB, file=dbfile) dbfile.close() def loadDbase(dbfilename=dbfilename): "parse data to reconstruct database" dbfile = open(dbfilename) import sys sys.stdin = dbfile db = {} key = input() while key != ENDDB: rec = {} field = input() while field != ENDREC: name, value = field.split(RECSEP) rec[name] = eval(value) field = input() db[key] = rec key = input() return db if __name__ == '__main__': from initdata import db storeDbase(db) This is a somewhat complex program, partly because it has both saving and loading logic and partly because it does its job the hard way; as we’ll see in a moment, there are better ways to get objects into files than by manually formatting and parsing them. For simple tasks, though, this does work; running Example 1-2 as a script writes the database out to a flat file. It has no printed output, but we can inspect the database file interactively after this script is run, either within IDLE or from a console window where Step 2: Storing Records Persistently | 17 you’re running these examples (as is, the database file shows up in the current working directory): ...\PP4E\Preview> python make_db_file.py ...\PP4E\Preview> python >>> for line in open('people-file'): ... print(line, end='') ... bob job=>'dev' pay=>30000 age=>42 name=>'Bob Smith' endrec. sue job=>'hdw' pay=>40000 age=>45 name=>'Sue Jones' endrec. tom job=>None pay=>0 age=>50 name=>'Tom' endrec. enddb. This file is simply our database’s content with added formatting. Its data originates from the test data initialization module we wrote in Example 1-1 because that is the module from which Example 1-2’s self-test code imports its data. In practice, Example 1-2 itself could be imported and used to store a variety of databases and files. Notice how data to be written is formatted with the as-code repr call and is re-created with the eval call, which treats strings as Python code. That allows us to store and recreate things like the None object, but it is potentially unsafe; you shouldn’t use eval if you can’t be sure that the database won’t contain malicious code. For our purposes, however, there’s probably no cause for alarm. Utility scripts To test further, Example 1-3 reloads the database from a file each time it is run. Example 1-3. PP4E\Preview\dump_db_file.py from make_db_file import loadDbase db = loadDbase() for key in db: print(key, '=>\n ', db[key]) print(db['sue']['name']) And Example 1-4 makes changes by loading, updating, and storing again. 18 | Chapter 1: A Sneak Preview Example 1-4. PP4E\Preview\update_db_file.py from make_db_file import loadDbase, storeDbase db = loadDbase() db['sue']['pay'] *= 1.10 db['tom']['name'] = 'Tom Tom' storeDbase(db) Here are the dump script and the update script in action at a system command line; both Sue’s pay and Tom’s name change between script runs. The main point to notice is that the data stays around after each script exits—our objects have become persistent simply because they are mapped to and from text files: ...\PP4E\Preview> python dump_db_file.py bob => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue => {'pay': 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} Sue Jones ...\PP4E\Preview> python update_db_file.py ...\PP4E\Preview> python dump_db_file.py bob => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue => {'pay': 44000.0, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom Tom'} Sue Jones As is, we’ll have to write Python code in scripts or at the interactive command line for each specific database update we need to perform (later in this chapter, we’ll do better by providing generalized console, GUI, and web-based interfaces instead). But at a basic level, our text file is a database of records. As we’ll learn in the next section, though, it turns out that we’ve just done a lot of pointless work. Using Pickle Files The formatted text file scheme of the prior section works, but it has some major limitations. For one thing, it has to read the entire database from the file just to fetch one record, and it must write the entire database back to the file after each set of updates. Although storing one record’s text per file would work around this limitation, it would also complicate the program further. For another thing, the text file approach assumes that the data separators it writes out to the file will not appear in the data to be stored: if the characters => happen to appear in the data, for example, the scheme will fail. We might work around this by generating XML text to represent records in the text file, using Python’s XML parsing tools, which we’ll meet later in this text, to reload; XML tags would avoid collisions with actual Step 2: Storing Records Persistently | 19 data’s text, but creating and parsing XML would complicate the program substantially too. Perhaps worst of all, the formatted text file scheme is already complex without being general: it is tied to the dictionary-of-dictionaries structure, and it can’t handle anything else without being greatly expanded. It would be nice if a general tool existed that could translate any sort of Python data to a format that could be saved in a file in a single step. That is exactly what the Python pickle module is designed to do. The pickle module translates an in-memory Python object into a serialized byte stream—a string of bytes that can be written to any file-like object. The pickle module also knows how to reconstruct the original object in memory, given the serialized byte stream: we get back the exact same object. In a sense, the pickle module replaces proprietary data formats —its serialized format is general and efficient enough for any program. With pickle, there is no need to manually translate objects to data when storing them persistently, and no need to manually parse a complex format to get them back. Pickling is similar in spirit to XML representations, but it’s both more Python-specific, and much simpler to code. The net effect is that pickling allows us to store and fetch native Python objects as they are and in a single step—we use normal Python syntax to process pickled records. Despite what it does, the pickle module is remarkably easy to use. Example 1-5 shows how to store our records in a flat file, using pickle. Example 1-5. PP4E\Preview\make_db_pickle.py from initdata import db import pickle dbfile = open('people-pickle', 'wb') pickle.dump(db, dbfile) dbfile.close() # use binary mode files in 3.X # data is bytes, not str When run, this script stores the entire database (the dictionary of dictionaries defined in Example 1-1) to a flat file named people-pickle in the current working directory. The pickle module handles the work of converting the object to a string. Example 1-6 shows how to access the pickled database after it has been created; we simply open the file and pass its content back to pickle to remake the object from its serialized string. Example 1-6. PP4E\Preview\dump_db_pickle.py import pickle dbfile = open('people-pickle', 'rb') db = pickle.load(dbfile) for key in db: print(key, '=>\n ', db[key]) print(db['sue']['name']) # use binary mode files in 3.X Here are these two scripts at work, at the system command line again; naturally, they can also be run in IDLE, and you can open and inspect the pickle file by running the same sort of code interactively as well: 20 | Chapter 1: A Sneak Preview ...\PP4E\Preview> python make_db_pickle.py ...\PP4E\Preview> python dump_db_pickle.py bob => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue => {'pay': 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} Sue Jones Updating with a pickle file is similar to a manually formatted file, except that Python is doing all of the formatting work for us. Example 1-7 shows how. Example 1-7. PP4E\Preview\update-db-pickle.py import pickle dbfile = open('people-pickle', 'rb') db = pickle.load(dbfile) dbfile.close() db['sue']['pay'] *= 1.10 db['tom']['name'] = 'Tom Tom' dbfile = open('people-pickle', 'wb') pickle.dump(db, dbfile) dbfile.close() Notice how the entire database is written back to the file after the records are changed in memory, just as for the manually formatted approach; this might become slow for very large databases, but we’ll ignore this for the moment. Here are our update and dump scripts in action—as in the prior section, Sue’s pay and Tom’s name change between scripts because they are written back to a file (this time, a pickle file): ...\PP4E\Preview> python update_db_pickle.py ...\PP4E\Preview> python dump_db_pickle.py bob => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue => {'pay': 44000.0, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom Tom'} Sue Jones As we’ll learn in Chapter 17, the Python pickling system supports nearly arbitrary object types—lists, dictionaries, class instances, nested structures, and more. There, we’ll also learn about the pickler’s text and binary storage protocols; as of Python 3, all protocols use bytes objects to represent pickled data, which in turn requires pickle files to be opened in binary mode for all protocols. As we’ll see later in this chapter, the pickler and its data format also underlie shelves and ZODB databases, and pickled class instances provide both data and behavior for objects stored. In fact, pickling is more general than these examples may imply. Because they accept any object that provides an interface compatible with files, pickling and unpickling may Step 2: Storing Records Persistently | 21 be used to transfer native Python objects to a variety of media. Using a network socket, for instance, allows us to ship pickled Python objects across a network and provides an alternative to larger protocols such as SOAP and XML-RPC. Using Per-Record Pickle Files As mentioned earlier, one potential disadvantage of this section’s examples so far is that they may become slow for very large databases: because the entire database must be loaded and rewritten to update a single record, this approach can waste time. We could improve on this by storing each record in the database in a separate flat file. The next three examples show one way to do so; Example 1-8 stores each record in its own flat file, using each record’s original key as its filename with a .pkl appended (it creates the files bob.pkl, sue.pkl, and tom.pkl in the current working directory). Example 1-8. PP4E\Preview\make_db_pickle_recs.py from initdata import bob, sue, tom import pickle for (key, record) in [('bob', bob), ('tom', tom), ('sue', sue)]: recfile = open(key + '.pkl', 'wb') pickle.dump(record, recfile) recfile.close() Next, Example 1-9 dumps the entire database by using the standard library’s glob module to do filename expansion and thus collect all the files in this directory with a .pkl extension. To load a single record, we open its file and deserialize with pickle; we must load only one record file, though, not the entire database, to fetch one record. Example 1-9. PP4E\Preview\dump_db_pickle_recs.py import pickle, glob for filename in glob.glob('*.pkl'): recfile = open(filename, 'rb') record = pickle.load(recfile) print(filename, '=>\n ', record) suefile = open('sue.pkl', 'rb') print(pickle.load(suefile)['name']) # for 'bob','sue','tom' # fetch sue's name Finally, Example 1-10 updates the database by fetching a record from its file, changing it in memory, and then writing it back to its pickle file. This time, we have to fetch and rewrite only a single record file, not the full database, to update. Example 1-10. PP4E\Preview\update_db_pickle_recs.py import pickle suefile = open('sue.pkl', 'rb') sue = pickle.load(suefile) suefile.close() 22 | Chapter 1: A Sneak Preview sue['pay'] *= 1.10 suefile = open('sue.pkl', 'wb') pickle.dump(sue, suefile) suefile.close() Here are our file-per-record scripts in action; the results are about the same as in the prior section, but database keys become real filenames now. In a sense, the filesystem becomes our top-level dictionary—filenames provide direct access to each record. ...\PP4E\Preview> python make_db_pickle_recs.py ...\PP4E\Preview> python dump_db_pickle_recs.py bob.pkl => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue.pkl => {'pay': 40000, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom.pkl => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} Sue Jones ...\PP4E\Preview> python update_db_pickle_recs.py ...\PP4E\Preview> python dump_db_pickle_recs.py bob.pkl => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue.pkl => {'pay': 44000.0, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom.pkl => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} Sue Jones Using Shelves Pickling objects to files, as shown in the preceding section, is an optimal scheme in many applications. In fact, some applications use pickling of Python objects across network sockets as a simpler alternative to network protocols such as the SOAP and XML-RPC web services architectures (also supported by Python, but much heavier than pickle). Moreover, assuming your filesystem can handle as many files as you’ll need, pickling one record per file also obviates the need to load and store the entire database for each update. If we really want keyed access to records, though, the Python standard library offers an even higher-level tool: shelves. Shelves automatically pickle objects to and from a keyed-access filesystem. They behave much like dictionaries that must be opened, and they persist after each program exits. Because they give us key-based access to stored records, there is no need to manually manage one flat file per record—the shelve system automatically splits up stored records and fetches and updates only those records that are accessed and changed. In this way, shelves provide utility similar to per-record pickle files, but they are usually easier to code. Step 2: Storing Records Persistently | 23 The shelve interface is just as simple as pickle: it is identical to dictionaries, with extra open and close calls. In fact, to your code, a shelve really does appear to be a persistent dictionary of persistent objects; Python does all the work of mapping its content to and from a file. For instance, Example 1-11 shows how to store our in-memory dictionary objects in a shelve for permanent keeping. Example 1-11. PP4E\Preview\make_db_shelve.py from initdata import bob, sue import shelve db = shelve.open('people-shelve') db['bob'] = bob db['sue'] = sue db.close() This script creates one or more files in the current directory with the name peopleshelve as a prefix (in Python 3.1 on Windows, people-shelve.bak, people-shelve.dat, and people-shelve.dir). You shouldn’t delete these files (they are your database!), and you should be sure to use the same base name in other scripts that access the shelve. Example 1-12, for instance, reopens the shelve and indexes it by key to fetch its stored records. Example 1-12. PP4E\Preview\dump_db_shelve.py import shelve db = shelve.open('people-shelve') for key in db: print(key, '=>\n ', db[key]) print(db['sue']['name']) db.close() We still have a dictionary of dictionaries here, but the top-level dictionary is really a shelve mapped onto a file. Much happens when you access a shelve’s keys—it uses pickle internally to serialize and deserialize objects stored, and it interfaces with a keyed-access filesystem. From your perspective, though, it’s just a persistent dictionary. Example 1-13 shows how to code shelve updates. Example 1-13. PP4E\Preview\update_db_shelve.py from initdata import tom import shelve db = shelve.open('people-shelve') sue = db['sue'] sue['pay'] *= 1.50 db['sue'] = sue db['tom'] = tom db.close() # fetch sue # update sue # add a new record Notice how this code fetches sue by key, updates in memory, and then reassigns to the key to update the shelve; this is a requirement of shelves by default, but not always of more advanced shelve-like systems such as ZODB, covered in Chapter 17. As we’ll see 24 | Chapter 1: A Sneak Preview later, shelve.open also has a newer writeback keyword argument, which, if passed True, causes all records loaded from the shelve to be cached in memory, and automatically written back to the shelve when it is closed; this avoids manual write backs on changes, but can consume memory and make closing slow. Also note how shelve files are explicitly closed. Although we don’t need to pass mode flags to shelve.open (by default it creates the shelve if needed, and opens it for reads and writes otherwise), some underlying keyed-access filesystems may require a close call in order to flush output buffers after changes. Finally, here are the shelve-based scripts on the job, creating, changing, and fetching records. The records are still dictionaries, but the database is now a dictionary-like shelve which automatically retains its state in a file between program runs: ...\PP4E\Preview> ...\PP4E\Preview> bob => {'pay': 30000, sue => {'pay': 40000, Sue Jones python make_db_shelve.py python dump_db_shelve.py 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} ...\PP4E\Preview> python update_db_shelve.py ...\PP4E\Preview> python dump_db_shelve.py bob => {'pay': 30000, 'job': 'dev', 'age': 42, 'name': 'Bob Smith'} sue => {'pay': 60000.0, 'job': 'hdw', 'age': 45, 'name': 'Sue Jones'} tom => {'pay': 0, 'job': None, 'age': 50, 'name': 'Tom'} Sue Jones When we ran the update and dump scripts here, we added a new record for key tom and increased Sue’s pay field by 50 percent. These changes are permanent because the record dictionaries are mapped to an external file by shelve. (In fact, this is a particularly good script for Sue—something she might consider scheduling to run often, using a cron job on Unix, or a Startup folder or msconfig entry on Windows…) What’s in a Name? Though it’s a surprisingly well-kept secret, Python gets its name from the 1970s British TV comedy series Monty Python’s Flying Circus. According to Python folklore, Guido van Rossum, Python’s creator, was watching reruns of the show at about the same time he needed a name for a new language he was developing. And as they say in show business, “the rest is history.” Because of this heritage, references to the comedy group’s work often show up in examples and discussion. For instance, the name Brian appears often in scripts; the words spam, lumberjack, and shrubbery have a special connotation to Python users; and presentations are sometimes referred to as The Spanish Inquisition. As a rule, if a Python user starts using phrases that have no relation to reality, they’re probably borrowed Step 2: Storing Records Persistently | 25 from the Monty Python series or movies. Some of these phrases might even pop up in this book. You don’t have to run out and rent The Meaning of Life or The Holy Grail to do useful work in Python, of course, but it can’t hurt. While “Python” turned out to be a distinctive name, it has also had some interesting side effects. For instance, when the Python newsgroup, comp.lang.python, came online in 1994, its first few weeks of activity were almost entirely taken up by people wanting to discuss topics from the TV show. More recently, a special Python supplement in the Linux Journal magazine featured photos of Guido garbed in an obligatory “nice red uniform.” Python’s news list still receives an occasional post from fans of the show. For instance, one early poster innocently offered to swap Monty Python scripts with other fans. Had he known the nature of the forum, he might have at least mentioned whether they were portable or not. Step 3: Stepping Up to OOP Let’s step back for a moment and consider how far we’ve come. At this point, we’ve created a database of records: the shelve, as well as per-record pickle file approaches of the prior section suffice for basic data storage tasks. As is, our records are represented as simple dictionaries, which provide easier-to-understand access to fields than do lists (by key, rather than by position). Dictionaries, however, still have some limitations that may become more critical as our program grows over time. For one thing, there is no central place for us to collect record processing logic. Extracting last names and giving raises, for instance, can be accomplished with code like the following: >>> import shelve >>> db = shelve.open('people-shelve') >>> bob = db['bob'] >>> bob['name'].split()[-1] 'Smith' >>> sue = db['sue'] >>> sue['pay'] *= 1.25 >>> sue['pay'] 75000.0 >>> db['sue'] = sue >>> db.close() # get bob's last name # give sue a raise This works, and it might suffice for some short programs. But if we ever need to change the way last names and raises are implemented, we might have to update this kind of code in many places in our program. In fact, even finding all such magical code snippets could be a challenge; hardcoding or cutting and pasting bits of logic redundantly like this in more than one place will almost always come back to haunt you eventually. It would be better to somehow hide—that is, encapsulate—such bits of code. Functions in a module would allow us to implement such operations in a single place and thus 26 | Chapter 1: A Sneak Preview avoid code redundancy, but still wouldn’t naturally associate them with the records themselves. What we’d like is a way to bind processing logic with the data stored in the database in order to make it easier to understand, debug, and reuse. Another downside to using dictionaries for records is that they are difficult to expand over time. For example, suppose that the set of data fields or the procedure for giving raises is different for different kinds of people (perhaps some people get a bonus each year and some do not). If we ever need to extend our program, there is no natural way to customize simple dictionaries. For future growth, we’d also like our software to support extension and customization in a natural way. If you’ve already studied Python in any sort of depth, you probably already know that this is where its OOP support begins to become attractive: Structure With OOP, we can naturally associate processing logic with record data—classes provide both a program unit that combines logic and data in a single package and a hierarchy that allows code to be easily factored to avoid redundancy. Encapsulation With OOP, we can also wrap up details such as name processing and pay increases behind method functions—i.e., we are free to change method implementations without breaking their users. Customization And with OOP, we have a natural growth path. Classes can be extended and customized by coding new subclasses, without changing or breaking already working code. That is, under OOP, we program by customizing and reusing, not by rewriting. OOP is an option in Python and, frankly, is sometimes better suited for strategic than for tactical tasks. It tends to work best when you have time for upfront planning—something that might be a luxury if your users have already begun storming the gates. But especially for larger systems that change over time, its code reuse and structuring advantages far outweigh its learning curve, and it can substantially cut development time. Even in our simple case, the customizability and reduced redundancy we gain from classes can be a decided advantage. Using Classes OOP is easy to use in Python, thanks largely to Python’s dynamic typing model. In fact, it’s so easy that we’ll jump right into an example: Example 1-14 implements our database records as class instances rather than as dictionaries. Example 1-14. PP4E\Preview\person_start.py class Person: def __init__(self, name, age, pay=0, job=None): Step 3: Stepping Up to OOP | 27 self.name self.age self.pay self.job = = = = name age pay job if __name__ == '__main__': bob = Person('Bob Smith', 42, 30000, 'software') sue = Person('Sue Jones', 45, 40000, 'hardware') print(bob.name, sue.pay) print(bob.name.split()[-1]) sue.pay *= 1.10 print(sue.pay) There is not much to this class—just a constructor method that fills out the instance with data passed in as arguments to the class name. It’s sufficient to represent a database record, though, and it can already provide tools such as defaults for pay and job fields that dictionaries cannot. The self-test code at the bottom of this file creates two instances (records) and accesses their attributes (fields); here is this file’s output when run under IDLE (a system command-line works just as well): Bob Smith 40000 Smith 44000.0 This isn’t a database yet, but we could stuff these objects into a list or dictionary as before in order to collect them as a unit: >>> from person_start import Person >>> bob = Person('Bob Smith', 42) >>> sue = Person('Sue Jones', 45, 40000) >>> people = [bob, sue] >>> for person in people: print(person.name, person.pay) # a "database" list Bob Smith 0 Sue Jones 40000 >>> x = [(person.name, person.pay) for person in people] >>> x [('Bob Smith', 0), ('Sue Jones', 40000)] >>> [rec.name for rec in people if rec.age >= 45] ['Sue Jones'] # SQL-ish query >>> [(rec.age ** 2 if rec.age >= 45 else rec.age) for rec in people] [42, 2025] Notice that Bob’s pay defaulted to zero this time because we didn’t pass in a value for that argument (maybe Sue is supporting him now?). We might also implement a class that represents the database, perhaps as a subclass of the built-in list or dictionary types, with insert and delete methods that encapsulate the way the database is implemented. We’ll abandon this path for now, though, because it will be more useful to store these 28 | Chapter 1: A Sneak Preview records persistently in a shelve, which already encapsulates stores and fetches behind an interface for us. Before we do, though, let’s add some logic. Adding Behavior So far, our class is just data: it replaces dictionary keys with object attributes, but it doesn’t add much to what we had before. To really leverage the power of classes, we need to add some behavior. By wrapping up bits of behavior in class method functions, we can insulate clients from changes. And by packaging methods in classes along with data, we provide a natural place for readers to look for code. In a sense, classes combine records and the programs that process those records; methods provide logic that interprets and updates the data (we say they are object-oriented, because they always process an object’s data). For instance, Example 1-15 adds the last-name and raise logic as class methods; methods use the self argument to access or update the instance (record) being processed. Example 1-15. PP4E\Preview\person.py class Person: def __init__(self, name, age, pay=0, job=None): self.name = name self.age = age self.pay = pay self.job = job def lastName(self): return self.name.split()[-1] def giveRaise(self, percent): self.pay *= (1.0 + percent) if __name__ == '__main__': bob = Person('Bob Smith', 42, 30000, 'software') sue = Person('Sue Jones', 45, 40000, 'hardware') print(bob.name, sue.pay) print(bob.lastName()) sue.giveRaise(.10) print(sue.pay) The output of this script is the same as the last, but the results are being computed by methods now, not by hardcoded logic that appears redundantly wherever it is required: Bob Smith 40000 Smith 44000.0 Adding Inheritance One last enhancement to our records before they become permanent: because they are implemented as classes now, they naturally support customization through the inheritance search mechanism in Python. Example 1-16, for instance, customizes the last Step 3: Stepping Up to OOP | 29 section’s Person class in order to give a 10 percent bonus by default to managers whenever they receive a raise (any relation to practice in the real world is purely coincidental). Example 1-16. PP4E\Preview\manager.py from person import Person class Manager(Person): def giveRaise(self, percent, bonus=0.1): self.pay *= (1.0 + percent + bonus) if __name__ == '__main__': tom = Manager(name='Tom Doe', age=50, pay=50000) print(tom.lastName()) tom.giveRaise(.20) print(tom.pay) When run, this script’s self-test prints the following: Doe 65000.0 Here, the Manager class appears in a module of its own, but it could have been added to the person module instead (Python doesn’t require just one class per file). It inherits the constructor and last-name methods from its superclass, but it customizes just the giveRaise method (there are a variety of ways to code this extension, as we’ll see later). Because this change is being added as a new subclass, the original Person class, and any objects generated from it, will continue working unchanged. Bob and Sue, for example, inherit the original raise logic, but Tom gets the custom version because of the class from which he is created. In OOP, we program by customizing, not by changing. In fact, code that uses our objects doesn’t need to be at all aware of what the raise method does—it’s up to the object to do the right thing based on the class from which it is created. As long as the object supports the expected interface (here, a method called giveRaise), it will be compatible with the calling code, regardless of its specific type, and even if its method works differently than others. If you’ve already studied Python, you may know this behavior as polymorphism; it’s a core property of the language, and it accounts for much of your code’s flexibility. When the following code calls the giveRaise method, for example, what happens depends on the obj object being processed; Tom gets a 20 percent raise instead of 10 percent because of the Manager class’s customization: >>> from person import Person >>> from manager import Manager >>> >>> >>> >>> bob = Person(name='Bob Smith', age=42, pay=10000) sue = Person(name='Sue Jones', age=45, pay=20000) tom = Manager(name='Tom Doe', age=55, pay=30000) db = [bob, sue, tom] 30 | Chapter 1: A Sneak Preview >>> for obj in db: obj.giveRaise(.10) # default or custom >>> for obj in db: print(obj.lastName(), '=>', obj.pay) Smith => 11000.0 Jones => 22000.0 Doe => 36000.0 Refactoring Code Before we move on, there are a few coding alternatives worth noting here. Most of these underscore the Python OOP model, and they serve as a quick review. Augmenting methods As a first alternative, notice that we have introduced some redundancy in Example 1-16: the raise calculation is now repeated in two places (in the two classes). We could also have implemented the customized Manager class by augmenting the inherited raise method instead of replacing it completely: class Manager(Person): def giveRaise(self, percent, bonus=0.1): Person.giveRaise(self, percent + bonus) The trick here is to call back the superclass’s version of the method directly, passing in the self argument explicitly. We still redefine the method, but we simply run the general version after adding 10 percent (by default) to the passed-in percentage. This coding pattern can help reduce code redundancy (the original raise method’s logic appears in only one place and so is easier to change) and is especially handy for kicking off superclass constructor methods in practice. If you’ve already studied Python OOP, you know that this coding scheme works because we can always call methods through either an instance or the class name. In general, the following are equivalent, and both forms may be used explicitly: instance.method(arg1, arg2) class.method(instance, arg1, arg2) In fact, the first form is mapped to the second—when calling through the instance, Python determines the class by searching the inheritance tree for the method name and passes in the instance automatically. Either way, within giveRaise, self refers to the instance that is the subject of the call. Display format For more object-oriented fun, we could also add a few operator overloading methods to our people classes. For example, a __str__ method, shown here, could return a string Step 3: Stepping Up to OOP | 31 to give the display format for our objects when they are printed as a whole—much better than the default display we get for an instance: class Person: def __str__(self): return '<%s => %s>' % (self.__class__.__name__, self.name) tom = Manager('Tom Jones', 50) print(tom) # prints: Tom Jones> Here __class__ gives the lowest class from which self was made, even though __str__ may be inherited. The net effect is that __str__ allows us to print instances directly instead of having to print specific attributes. We could extend this __str__ to loop through the instance’s __dict__ attribute dictionary to display all attributes generically; for this preview we’ll leave this as a suggested exercise. We might even code an __add__ method to make + expressions automatically call the giveRaise method. Whether we should is another question; the fact that a + expression gives a person a raise might seem more magical to the next person reading our code than it should. Constructor customization Finally, notice that we didn’t pass the job argument when making a manager in Example 1-16; if we had, it would look like this with keyword arguments: tom = Manager(name='Tom Doe', age=50, pay=50000, job='manager') The reason we didn’t include a job in the example is that it’s redundant with the class of the object: if someone is a manager, their class should imply their job title. Instead of leaving this field blank, though, it may make more sense to provide an explicit constructor for managers, which fills in this field automatically: class Manager(Person): def __init__(self, name, age, pay): Person.__init__(self, name, age, pay, 'manager') Now when a manager is created, its job is filled in automatically. The trick here is to call to the superclass’s version of the method explicitly, just as we did for the giveRaise method earlier in this section; the only difference here is the unusual name for the constructor method. Alternative classes We won’t use any of this section’s three extensions in later examples, but to demonstrate how they work, Example 1-17 collects these ideas in an alternative implementation of our Person classes. 32 | Chapter 1: A Sneak Preview Example 1-17. PP4E\Preview\person_alternative.py """ Alternative implementation of person classes, with data, behavior, and operator overloading (not used for objects stored persistently) """ class Person: """ a general person: data+logic """ def __init__(self, name, age, pay=0, job=None): self.name = name self.age = age self.pay = pay self.job = job def lastName(self): return self.name.split()[-1] def giveRaise(self, percent): self.pay *= (1.0 + percent) def __str__(self): return ('<%s => %s: %s, %s>' % (self.__class__.__name__, self.name, self.job, self.pay)) class Manager(Person): """ a person with custom raise inherits general lastname, str """ def __init__(self, name, age, pay): Person.__init__(self, name, age, pay, 'manager') def giveRaise(self, percent, bonus=0.1): Person.giveRaise(self, percent + bonus) if __name__ == '__main__': bob = Person('Bob Smith', 44) sue = Person('Sue Jones', 47, 40000, 'hardware') tom = Manager(name='Tom Doe', age=50, pay=50000) print(sue, sue.pay, sue.lastName()) for obj in (bob, sue, tom): obj.giveRaise(.10) # run this obj's giveRaise print(obj) # run common __str__ method Notice the polymorphism in this module’s self-test loop: all three objects share the constructor, last-name, and printing methods, but the raise method called is dependent upon the class from which an instance is created. When run, Example 1-17 prints the following to standard output—the manager’s job is filled in at construction, we get the new custom display format for our objects, and the new version of the manager’s raise method works as before: Sue Jones: hardware, 40000> 40000 Jones Bob Smith: None, 0.0> Sue Jones: hardware, 44000.0> Tom Doe: manager, 60000.0> Step 3: Stepping Up to OOP | 33 Such refactoring (restructuring) of code is common as class hierarchies grow and evolve. In fact, as is, we still can’t give someone a raise if his pay is zero (Bob is out of luck); we probably need a way to set pay, too, but we’ll leave such extensions for the next release. The good news is that Python’s flexibility and readability make refactoring easy—it’s simple and quick to restructure your code. If you haven’t used the language yet, you’ll find that Python development is largely an exercise in rapid, incremental, and interactive programming, which is well suited to the shifting needs of real-world projects. Adding Persistence It’s time for a status update. We now have encapsulated in the form of classes customizable implementations of our records and their processing logic. Making our classbased records persistent is a minor last step. We could store them in per-record pickle files again; a shelve-based storage medium will do just as well for our goals and is often easier to code. Example 1-18 shows how. Example 1-18. PP4E\Preview\make_db_classes.py import shelve from person import Person from manager import Manager bob = Person('Bob Smith', 42, 30000, 'software') sue = Person('Sue Jones', 45, 40000, 'hardware') tom = Manager('Tom Doe', 50, 50000) db = shelve.open('class-shelve') db['bob'] = bob db['sue'] = sue db['tom'] = tom db.close() This file creates three class instances (two from the original class and one from its customization) and assigns them to keys in a newly created shelve file to store them permanently. In other words, it creates a shelve of class instances; to our code, the database looks just like a dictionary of class instances, but the top-level dictionary is mapped to a shelve file again. To check our work, Example 1-19 reads the shelve and prints fields of its records. Example 1-19. PP4E\Preview\dump_db_classes.py import shelve db = shelve.open('class-shelve') for key in db: print(key, '=>\n ', db[key].name, db[key].pay) bob = db['bob'] print(bob.lastName()) print(db['tom'].lastName()) 34 | Chapter 1: A Sneak Preview Note that we don’t need to reimport the Person class here in order to fetch its instances from the shelve or run their methods. When instances are shelved or pickled, the underlying pickling system records both instance attributes and enough information to locate their classes automatically when they are later fetched (the class’s module simply has to be on the module search path when an instance is loaded). This is on purpose; because the class and its instances in the shelve are stored separately, you can change the class to modify the way stored instances are interpreted when loaded (more on this later in the book). Here is the shelve dump script’s output just after creating the shelve with the maker script: bob => Bob Smith 30000 sue => Sue Jones 40000 tom => Tom Doe 50000 Smith Doe As shown in Example 1-20, database updates are as simple as before (compare this to Example 1-13), but dictionary keys become attributes of instance objects, and updates are implemented by class method calls instead of hardcoded logic. Notice how we still fetch, update, and reassign to keys to update the shelve. Example 1-20. PP4E\Preview\update_db_classes.py import shelve db = shelve.open('class-shelve') sue = db['sue'] sue.giveRaise(.25) db['sue'] = sue tom = db['tom'] tom.giveRaise(.20) db['tom'] = tom db.close() And last but not least, here is the dump script again after running the update script; Tom and Sue have new pay values, because these objects are now persistent in the shelve. We could also open and inspect the shelve by typing code at Python’s interactive command line; despite its longevity, the shelve is just a Python object containing Python objects. bob => Bob Smith 30000 sue => Sue Jones 50000.0 tom => Tom Doe 65000.0 Smith Doe Step 3: Stepping Up to OOP | 35 Tom and Sue both get a raise this time around, because they are persistent objects in the shelve database. Although shelves can also store simpler object types such as lists and dictionaries, class instances allow us to combine both data and behavior for our stored items. In a sense, instance attributes and class methods take the place of records and processing programs in more traditional schemes. Other Database Options At this point, we have a full-fledged database system: our classes simultaneously implement record data and record processing, and they encapsulate the implementation of the behavior. And the Python pickle and shelve modules provide simple ways to store our database persistently between program executions. This is not a relational database (we store objects, not tables, and queries take the form of Python object processing code), but it is sufficient for many kinds of programs. If we need more functionality, we could migrate this application to even more powerful tools. For example, should we ever need full-blown SQL query support, there are interfaces that allow Python scripts to communicate with relational databases such as MySQL, PostgreSQL, and Oracle in portable ways. ORMs (object relational mappers) such as SQLObject and SqlAlchemy offer another approach which retains the Python class view, but translates it to and from relational database tables—in a sense providing the best of both worlds, with Python class syntax on top, and enterprise-level databases underneath. Moreover, the open source ZODB system provides a more comprehensive object database for Python, with support for features missing in shelves, including concurrent updates, transaction commits and rollbacks, automatic updates on in-memory component changes, and more. We’ll explore these more advanced third-party tools in Chapter 17. For now, let’s move on to putting a good face on our system. “Buses Considered Harmful” Over the years, Python has been remarkably well supported by the volunteer efforts of both countless individuals and formal organizations. Today, the nonprofit Python Software Foundation (PSF) oversees Python conferences and other noncommercial activities. The PSF was preceded by the PSA, a group that was originally formed in response to an early thread on the Python newsgroup that posed the semiserious question: “What would happen if Guido was hit by a bus?” These days, Python creator Guido van Rossum is still the ultimate arbiter of proposed Python changes. He was officially anointed the BDFL—Benevolent Dictator for Life— of Python at the first Python conference and still makes final yes and no decisions on language changes (and apart from 3.0’s deliberate incompatibilities, has usually said no: a good thing in the programming languages domain, because Python tends to change slowly and in backward-compatible ways). 36 | Chapter 1: A Sneak Preview But Python’s user base helps support the language, work on extensions, fix bugs, and so on. It is a true community project. In fact, Python development is now a completely open process—anyone can inspect the latest source code files or submit patches by visiting a website (see http://www.python.org for details). As an open source package, Python development is really in the hands of a very large cast of developers working in concert around the world—so much so that if the BDFL ever does pass the torch, Python will almost certainly continue to enjoy the kind of support its users have come to expect. Though not without pitfalls of their own, open source projects by nature tend to reflect the needs of their user communities more than either individuals or shareholders. Given Python’s popularity, bus attacks seem less threatening now than they once did. Of course, I can’t speak for Guido. Step 4: Adding Console Interaction So far, our database program consists of class instances stored in a shelve file, as coded in the preceding section. It’s sufficient as a storage medium, but it requires us to run scripts from the command line or type code interactively in order to view or process its content. Improving on this is straightforward: simply code more general programs that interact with users, either from a console window or from a full-blown graphical interface. A Console Shelve Interface Let’s start with something simple. The most basic kind of interface we can code would allow users to type keys and values in a console window in order to process the database (instead of writing Python program code). Example 1-21, for instance, implements a simple interactive loop that allows a user to query multiple record objects in the shelve by key. Example 1-21. PP4E\Preview\peopleinteract_query.py # interactive queries import shelve fieldnames = ('name', 'age', 'job', 'pay') maxfield = max(len(f) for f in fieldnames) db = shelve.open('class-shelve') while True: key = input('\nKey? => ') if not key: break try: record = db[key] # key or empty line, exc at eof # fetch by key, show in console Step 4: Adding Console Interaction | 37 except: print('No such key "%s"!' % key) else: for field in fieldnames: print(field.ljust(maxfield), '=>', getattr(record, field)) This script uses the getattr built-in function to fetch an object’s attribute when given its name string, and the ljust left-justify method of strings to align outputs (max field, derived from a generator expression, is the length of the longest field name). When run, this script goes into a loop, inputting keys from the interactive user (technically, from the standard input stream, which is usually a console window) and displaying the fetched records field by field. An empty line ends the session. If our shelve of class instances is still in the state we left it near the end of the last section: ...\PP4E\Preview> dump_db_classes.py bob => Bob Smith 30000 sue => Sue Jones 50000.0 tom => Tom Doe 65000.0 Smith Doe We can then use our new script to query the object database interactively, by key: ...\PP4E\Preview> peopleinteract_query.py Key? name age job pay => => => => => sue Sue Jones 45 hardware 50000.0 Key? => nobody No such key "nobody"! Key? => Example 1-22 goes further and allows interactive updates. For an input key, it inputs values for each field and either updates an existing record or creates a new object and stores it under the key. Example 1-22. PP4E\Preview\peopleinteract_update.py # interactive updates import shelve from person import Person fieldnames = ('name', 'age', 'job', 'pay') db = shelve.open('class-shelve') while True: key = input('\nKey? => ') if not key: break 38 | Chapter 1: A Sneak Preview if key in db: record = db[key] # update existing record else: # or make/store new rec record = Person(name='?', age='?') # eval: quote strings for field in fieldnames: currval = getattr(record, field) newtext = input('\t[%s]=%s\n\t\tnew?=>' % (field, currval)) if newtext: setattr(record, field, eval(newtext)) db[key] = record db.close() Notice the use of eval in this script to convert inputs (as usual, that allows any Python object type, but it means you must quote string inputs explicitly) and the use of setattr call to assign an attribute given its name string. When run, this script allows any number of records to be added and changed; to keep the current value of a record’s field, press the Enter key when prompted for a new value: Key? => tom [name]=Tom Doe new?=> [age]=50 new?=>56 [job]=None new?=>'mgr' [pay]=65000.0 new?=>90000 Key? => nobody [name]=? new?=>'John Doh' [age]=? new?=>55 [job]=None new?=> [pay]=0 new?=>None Key? => This script is still fairly simplistic (e.g., errors aren’t handled), but using it is much easier than manually opening and modifying the shelve at the Python interactive prompt, especially for nonprogrammers. Run the query script to check your work after an update (we could combine query and update into a single script if this becomes too cumbersome, albeit at some cost in code and user-experience complexity): Key? name age job pay => => => => => tom Tom Doe 56 mgr 90000 Key? => nobody name => John Doh Step 4: Adding Console Interaction | 39 age job pay => 55 => None => None Key? => Step 5: Adding a GUI The console-based interface approach of the preceding section works, and it may be sufficient for some users assuming that they are comfortable with typing commands in a console window. With just a little extra work, though, we can add a GUI that is more modern, easier to use, less error prone, and arguably sexier. GUI Basics As we’ll see later in this book, a variety of GUI toolkits and builders are available for Python programmers: tkinter, wxPython, PyQt, PythonCard, Dabo, and more. Of these, tkinter ships with Python, and it is something of a de facto standard. tkinter is a lightweight toolkit and so meshes well with a scripting language such as Python; it’s easy to do basic things with tkinter, and it’s straightforward to do more advanced things with extensions and OOP-based code. As an added bonus, tkinter GUIs are portable across Windows, Linux/Unix, and Macintosh; simply copy the source code to the machine on which you wish to use your GUI. tkinter doesn’t come with all the bells and whistles of larger toolkits such as wxPython or PyQt, but that’s a major factor behind its relative simplicity, and it makes it ideal for getting started in the GUI domain. Because tkinter is designed for scripting, coding GUIs with it is straightforward. We’ll study all of its concepts and tools later in this book. But as a first example, the first program in tkinter is just a few lines of code, as shown in Example 1-23. Example 1-23. PP4E\Preview\tkinter001.py from tkinter import * Label(text='Spam').pack() mainloop() From the tkinter module (really, a module package in Python 3), we get screen device (a.k.a. “widget”) construction calls such as Label; geometry manager methods such as pack; widget configuration presets such as the TOP and RIGHT attachment side hints we’ll use later for pack; and the mainloop call, which starts event processing. This isn’t the most useful GUI ever coded, but it demonstrates tkinter basics and it builds the fully functional window shown in Figure 1-1 in just three simple lines of code. Its window is shown here, like all GUIs in this book, running on Windows 7; it works the same on other platforms (e.g., Mac OS X, Linux, and older versions of Windows), but renders in with native look and feel on each. 40 | Chapter 1: A Sneak Preview Figure 1-1. tkinter001.py window You can launch this example in IDLE, from a console command line, or by clicking its icon—the same way you can run other Python scripts. tkinter itself is a standard part of Python and works out-of-the-box on Windows and others, though you may need extra configuration or install steps on some computers (more details later in this book). It’s not much more work to code a GUI that actually responds to a user: Example 1-24 implements a GUI with a button that runs the reply function each time it is pressed. Example 1-24. PP4E\Preview\ tkinter101.py from tkinter import * from tkinter.messagebox import showinfo def reply(): showinfo(title='popup', message='Button pressed!') window = Tk() button = Button(window, text='press', command=reply) button.pack() window.mainloop() This example still isn’t very sophisticated—it creates an explicit Tk main window for the application to serve as the parent container of the button, and it builds the simple window shown in Figure 1-2 (in tkinter, containers are passed in as the first argument when making a new widget; they default to the main window). But this time, each time you click the “press” button, the program responds by running Python code that pops up the dialog window in Figure 1-3. Figure 1-2. tkinter101.py main window Notice that the pop-up dialog looks like it should for Windows 7, the platform on which this screenshot was taken; again, tkinter gives us a native look and feel that is appropriate for the machine on which it is running. We can customize this GUI in many ways (e.g., by changing colors and fonts, setting window titles and icons, using photos Step 5: Adding a GUI | 41 Figure 1-3. tkinter101.py common dialog pop up on buttons instead of text), but part of the power of tkinter is that we need to set only the options we are interested in tailoring. Using OOP for GUIs All of our GUI examples so far have been top-level script code with a function for handling events. In larger programs, it is often more useful to code a GUI as a subclass of the tkinter Frame widget—a container for other widgets. Example 1-25 shows our single-button GUI recoded in this way as a class. Example 1-25. PP4E\Preview\tkinter102.py from tkinter import * from tkinter.messagebox import showinfo class MyGui(Frame): def __init__(self, parent=None): Frame.__init__(self, parent) button = Button(self, text='press', command=self.reply) button.pack() def reply(self): showinfo(title='popup', message='Button pressed!') if __name__ == '__main__': window = MyGui() window.pack() window.mainloop() The button’s event handler is a bound method—self.reply, an object that remembers both self and reply when later called. This example generates the same window and pop up as Example 1-24 (Figures 1-2 and 1-3); but because it is now a subclass of Frame, it automatically becomes an attachable component—i.e., we can add all of the widgets this class creates, as a package, to any other GUI, just by attaching this Frame to the GUI. Example 1-26 shows how. 42 | Chapter 1: A Sneak Preview Example 1-26. PP4E\Preview\attachgui.py from tkinter import * from tkinter102 import MyGui # main app window mainwin = Tk() Label(mainwin, text=__name__).pack() # popup window popup = Toplevel() Label(popup, text='Attach').pack(side=LEFT) MyGui(popup).pack(side=RIGHT) mainwin.mainloop() # attach my frame This example attaches our one-button GUI to a larger window, here a Toplevel popup window created by the importing application and passed into the construction call as the explicit parent (you will also get a Tk main window; as we’ll learn later, you always do, whether it is made explicit in your code or not). Our one-button widget package is attached to the right side of its container this time. If you run this live, you’ll get the scene captured in Figure 1-4; the “press” button is our attached custom Frame. Figure 1-4. Attaching GUIs Moreover, because MyGui is coded as a class, the GUI can be customized by the usual inheritance mechanism; simply define a subclass that replaces the parts that differ. The reply method, for example, can be customized this way to do something unique, as demonstrated in Example 1-27. Example 1-27. PP4E\Preview\customizegui.py from tkinter import mainloop from tkinter.messagebox import showinfo from tkinter102 import MyGui Step 5: Adding a GUI | 43 class CustomGui(MyGui): def reply(self): showinfo(title='popup', message='Ouch!') # inherit init # replace reply if __name__ == '__main__': CustomGui().pack() mainloop() When run, this script creates the same main window and button as the original MyGui class. But pressing its button generates a different reply, as shown in Figure 1-5, because the custom version of the reply method runs. Figure 1-5. Customizing GUIs Although these are still small GUIs, they illustrate some fairly large ideas. As we’ll see later in the book, using OOP like this for inheritance and attachment allows us to reuse packages of widgets in other programs—calculators, text editors, and the like can be customized and added as components to other GUIs easily if they are classes. As we’ll also find, subclasses of widget class can provide a common appearance or standardized behavior for all their instances—similar in spirit to what some observers might call GUI styles or themes. It’s a normal byproduct of Python and OOP. Getting Input from a User As a final introductory script, Example 1-28 shows how to input data from the user in an Entry widget and display it in a pop-up dialog. The lambda it uses defers the call to the reply function so that inputs can be passed in—a common tkinter coding pattern; without the lambda, reply would be called when the button is made, instead of when it is later pressed (we could also use ent as a global variable within reply, but that makes it less general). This example also demonstrates how to change the icon and title of a top-level window; here, the window icon file is located in the same directory as the script (if the icon call in this script fails on your platform, try commenting-out the call; icons are notoriously platform specific). 44 | Chapter 1: A Sneak Preview Example 1-28. PP4E\Preview\tkinter103.py from tkinter import * from tkinter.messagebox import showinfo def reply(name): showinfo(title='Reply', message='Hello %s!' % name) top = Tk() top.title('Echo') top.iconbitmap('py-blue-trans-out.ico') Label(top, text="Enter your name:").pack(side=TOP) ent = Entry(top) ent.pack(side=TOP) btn = Button(top, text="Submit", command=(lambda: reply(ent.get()))) btn.pack(side=LEFT) top.mainloop() As is, this example is just three widgets attached to the Tk main top-level window; later we’ll learn how to use nested Frame container widgets in a window like this to achieve a variety of layouts for its three widgets. Figure 1-6 gives the resulting main and popup windows after the Submit button is pressed. We’ll see something very similar later in this chapter, but rendered in a web browser with HTML. Figure 1-6. Fetching input from a user The code we’ve seen so far demonstrates many of the core concepts in GUI programming, but tkinter is much more powerful than these examples imply. There are more than 20 widgets in tkinter and many more ways to input data from a user, including multiple-line text, drawing canvases, pull-down menus, radio and check buttons, and scroll bars, as well as other layout and event handling mechanisms. Beyond tkinter itself, both open source extensions such as PMW, as well as the Tix and ttk toolkits now part of Python’s standard library, can add additional widgets we can use in our Step 5: Adding a GUI | 45 Python tkinter GUIs and provide an even more professional look and feel. To hint at what is to come, let’s put tkinter to work on our database of people. A GUI Shelve Interface For our database application, the first thing we probably want is a GUI for viewing the stored data—a form with field names and values—and a way to fetch records by key. It would also be useful to be able to update a record with new field values given its key and to add new records from scratch by filling out the form. To keep this simple, we’ll use a single GUI for all of these tasks. Figure 1-7 shows the window we are going to code as it looks in Windows 7; the record for the key sue has been fetched and displayed (our shelve is as we last left it again). This record is really an instance of our class in our shelve file, but the user doesn’t need to care. Figure 1-7. peoplegui.py main display/input window Coding the GUI Also, to keep this simple, we’ll assume that all records in the database have the same sets of fields. It would be a minor extension to generalize this for any set of fields (and come up with a general form GUI constructor tool in the process), but we’ll defer such evolutions to later in this book. Example 1-29 implements the GUI shown in Figure 1-7. Example 1-29. PP4E\Preview\peoplegui.py """ Implement a GUI for viewing and updating class instances stored in a shelve; the shelve lives on the machine this script runs on, as 1 or more local files; """ from tkinter import * from tkinter.messagebox import showerror import shelve shelvename = 'class-shelve' fieldnames = ('name', 'age', 'job', 'pay') 46 | Chapter 1: A Sneak Preview def makeWidgets(): global entries window = Tk() window.title('People Shelve') form = Frame(window) form.pack() entries = {} for (ix, label) in enumerate(('key',) + fieldnames): lab = Label(form, text=label) ent = Entry(form) lab.grid(row=ix, column=0) ent.grid(row=ix, column=1) entries[label] = ent Button(window, text="Fetch", command=fetchRecord).pack(side=LEFT) Button(window, text="Update", command=updateRecord).pack(side=LEFT) Button(window, text="Quit", command=window.quit).pack(side=RIGHT) return window def fetchRecord(): key = entries['key'].get() try: record = db[key] # fetch by key, show in GUI except: showerror(title='Error', message='No such key!') else: for field in fieldnames: entries[field].delete(0, END) entries[field].insert(0, repr(getattr(record, field))) def updateRecord(): key = entries['key'].get() if key in db: record = db[key] # update existing record else: from person import Person # make/store new one for key record = Person(name='?', age='?') # eval: strings must be quoted for field in fieldnames: setattr(record, field, eval(entries[field].get())) db[key] = record db = shelve.open(shelvename) window = makeWidgets() window.mainloop() db.close() # back here after quit or window close This script uses the widget grid method to arrange labels and entries, instead of pack; as we’ll see later, gridding arranges by rows and columns, and so it is a natural for forms that horizontally align labels with entries well. We’ll also see later that forms can usually be laid out just as nicely using pack with nested row frames and fixed-width labels. Although the GUI doesn’t handle window resizes well yet (that requires configuration options we’ll explore later), adding this makes the grid and pack alternatives roughly the same in code size. Step 5: Adding a GUI | 47 Notice how the end of this script opens the shelve as a global variable and starts the GUI; the shelve remains open for the lifespan of the GUI (mainloop returns only after the main window is closed). As we’ll see in the next section, this state retention is very different from the web model, where each interaction is normally a standalone program. Also notice that the use of global variables makes this code simple but unusable outside the context of our database; more on this later. Using the GUI The GUI we’re building is fairly basic, but it provides a view on the shelve file and allows us to browse and update the file without typing any code. To fetch a record from the shelve and display it on the GUI, type its key into the GUI’s “key” field and click Fetch. To change a record, type into its input fields after fetching it and click Update; the values in the GUI will be written to the record in the database. And to add a new record, fill out all of the GUI’s fields with new values and click Update—the new record will be added to the shelve file using the key and field inputs you provide. In other words, the GUI’s fields are used for both display and input. Figure 1-8 shows the scene after adding a new record (via Update), and Figure 1-9 shows an error dialog pop up issued when users try to fetch a key that isn’t present in the shelve. Figure 1-8. peoplegui.py after adding a new persistent object Figure 1-9. peoplegui.py common error dialog pop up 48 | Chapter 1: A Sneak Preview Notice how we’re using repr again to display field values fetched from the shelve and eval to convert field values to Python objects before they are stored in the shelve. As mentioned previously, this is potentially dangerous if someone sneaks some malicious code into our shelve, but we’ll finesse such concerns for now. Keep in mind, though, that this scheme means that strings must be quoted in input fields other than the key—they are assumed to be Python code. In fact, you could type an arbitrary Python expression in an input field to specify a value for an update. Typing "Tom"*3 in the name field, for instance, would set the name to TomTomTom after an update (for better or worse!); fetch to see the result. Even though we now have a GUI for browsing and changing records, we can still check our work by interactively opening and inspecting the shelve file or by running scripts such as the dump utility in Example 1-19. Remember, despite the fact that we’re now viewing records in a GUI’s windows, the database is a Python shelve file containing native Python class instance objects, so any Python code can access it. Here is the dump script at work after adding and changing a few persistent objects in the GUI: ...\PP4E\Preview> python dump_db_classes.py sue => Sue Jones 50000.0 bill => bill 9999 nobody => John Doh None tomtom => Tom Tom 40000 tom => Tom Doe 90000 bob => Bob Smith 30000 peg => 1 4 Smith Doe Future directions Although this GUI does the job, there is plenty of room for improvement: • As coded, this GUI is a simple set of functions that share the global list of input fields (entries) and a global shelve (db). We might instead pass db in to makeWidgets, and pass along both these two objects as function arguments to the callback handlers using the lambda trick of the prior section. Though not crucial in a script this small, as a rule of thumb, making your external dependencies explicit like this makes your code both easier to understand and reusable in other contexts. • We could also structure this GUI as a class to support attachment and customization (globals would become instance attributes), though it’s unlikely that we’ll need to reuse such a specific GUI. Step 5: Adding a GUI | 49 • More usefully, we could pass in the fieldnames tuple as an input parameter to the functions here to allow them to be used for other record types in the future. Code at the bottom of the file would similarly become a function with a passed-in shelve filename, and we would also need to pass in a new record construction call to the update function because Person could not be hardcoded. Such generalization is beyond the scope of this preview, but it makes for a nice exercise if you are so inclined. Later, I’ll also point you to a suggested reading example in the book examples package, PyForm, which takes a different approach to generalized form construction. • To make this GUI more user friendly, it might also be nice to add an index window that displays all the keys in the database in order to make browsing easier. Some sort of verification before updates might be useful as well, and Delete and Clear buttons would be simple to code. Furthermore, assuming that inputs are Python code may be more bother than it is worth; a simpler input scheme might be easier and safer. (I won’t officially say these are suggested exercises too, but it sounds like they could be.) • We could also support window resizing (as we’ll learn, widgets can grow and shrink with the window) and provide an interface for calling methods available on stored instances’ classes too (as is, the pay field can be updated, but there is no way to invoke the giveRaise method). • If we plan to distribute this GUI widely, we might package it up as a standalone executable program—a frozen binary in Python terminology—using third-party tools such as Py2Exe, PyInstaller, and others (search the Web for pointers). Such a program can be run directly without installing Python on the receiving end, because the Python bytecode interpreter is included in the executable itself. I’ll leave all such extensions as points to ponder, and revisit some of them later in this book. Before we move on, two notes. First, I should mention that even more graphical packages are available to Python programmers. For instance, if you need to do graphics beyond basic windows, the tkinter Canvas widget supports freeform graphics. Thirdparty extensions such as Blender, OpenGL, VPython, PIL, VTK, Maya, and PyGame provide even more advanced graphics, visualization, and animation tools for use with Python scripts. Moreover, the PMW, Tix, and ttk widget kits mentioned earlier extend tkinter itself. See Python’s library manual for Tix and ttk, and try the PyPI site or a web search for third-party graphics extensions. And in deference to fans of other GUI toolkits such as wxPython and PyQt, I should also note that there are other GUI options to choose from and that choice is sometimes very subjective. tkinter is shown here because it is mature, robust, fully open source, well documented, well supported, lightweight, and a standard part of Python. By most accounts, it remains the standard for building portable GUIs in Python. 50 | Chapter 1: A Sneak Preview Other GUI toolkits for Python have pros and cons of their own, discussed later in this book. For example, some exchange code simplicity for richer widget sets. wxPython, for example, is much more feature-rich, but it’s also much more complicated to use. By and large, though, other toolkits are variations on a theme—once you’ve learned one GUI toolkit, others are easy to pick up. Because of that, we’ll focus on learning one toolkit in its entirety in this book instead of sampling many partially. Although they are free to employ network access at will, programs written with traditional GUIs like tkinter generally run on a single, self-contained machine. Some consider web pages to be a kind of GUI as well, but you’ll have to read the next and final section of this chapter to judge that for yourself. For a Good Time… There’s much more to the tkinter toolkit than we’ve touched on in this preview, of course, and we’ll study it in depth in this book. As another quick example to hint at what’s possible, though, the following script, fungui.py, uses the Python random module to pick from a list, makes new independent windows with Toplevel, and uses the tkinter after callback to loop by scheduling methods to run again after a number of milliseconds: from tkinter import * import random fontsize = 30 colors = ['red', 'green', 'blue', 'yellow', 'orange', 'cyan', 'purple'] def onSpam(): popup = Toplevel() color = random.choice(colors) Label(popup, text='Popup', bg='black', fg=color).pack(fill=BOTH) mainLabel.config(fg=color) def onFlip(): mainLabel.config(fg=random.choice(colors)) main.after(250, onFlip) def onGrow(): global fontsize fontsize += 5 mainLabel.config(font=('arial', fontsize, 'italic')) main.after(100, onGrow) main = Tk() mainLabel = Label(main, text='Fun Gui!', relief=RAISED) mainLabel.config(font=('arial', fontsize, 'italic'), fg='cyan',bg='navy') mainLabel.pack(side=TOP, expand=YES, fill=BOTH) Button(main, text='spam', command=onSpam).pack(fill=X) Button(main, text='flip', command=onFlip).pack(fill=X) Button(main, text='grow', command=onGrow).pack(fill=X) main.mainloop() Run this on your own to see how it works. It creates a main window with a custom label and three buttons—one button pops up a new window with a randomly colored label, and the other two kick off potentially independent timer loops, one of which Step 5: Adding a GUI | 51 keeps changing the color used in the main window, and another that keeps expanding the main window label’s font. Be careful if you do run this, though; the colors flash, and the label font gets bigger 10 times per second, so be sure you are able to kill the main window before it gets away from you. Hey—I warned you! Step 6: Adding a Web Interface GUI interfaces are easier to use than command lines and are often all we need to simplify access to data. By making our database available on the Web, though, we can open it up to even wider use. Anyone with Internet access and a web browser can access the data, regardless of where they are located and which machine they are using. Anything from workstations to cell phones will suffice. Moreover, web-based interfaces require only a web browser; there is no need to install Python to access the data except on the single-server machine. Although traditional web-based approaches may sacrifice some of the utility and speed of in-process GUI toolkits, their portability gain can be compelling. As we’ll also see later in this book, there are a variety of ways to go about scripting interactive web pages of the sort we’ll need in order to access our data. Basic serverside CGI scripting is more than adequate for simple tasks like ours. Because it’s perhaps the simplest approach, and embodies the foundations of more advanced techniques, CGI scripting is also well-suited to getting started on the Web. For more advanced applications, a wealth of toolkits and frameworks for Python— including Django, TurboGears, Google’s App Engine, pylons, web2py, Zope, Plone, Twisted, CherryPy, Webware, mod_python, PSP, and Quixote—can simplify common tasks and provide tools that we might otherwise need to code from scratch in the CGI world. Though they pose a new set of tradeoffs, emerging technologies such as Flex, Silverlight, and pyjamas (an AJAX-based port of the Google Web Toolkit to Python, and Python-to-JavaScript compiler) offer additional paths to achieving interactive or dynamic user-interfaces in web pages on clients, and open the door to using Python in Rich Internet Applications (RIAs). I’ll say more about these tools later. For now, let’s keep things simple and code a CGI script. CGI Basics CGI scripting in Python is easy as long as you already have a handle on things like HTML forms, URLs, and the client/server model of the Web (all topics we’ll address in detail later in this book). Whether you’re aware of all the underlying details or not, the basic interaction model is probably familiar. In a nutshell, a user visits a website and receives a form, coded in HTML, to be filled out in his or her browser. After submitting the form, a script, identified within either 52 | Chapter 1: A Sneak Preview the form or the address used to contact the server, is run on the server and produces another HTML page as a reply. Along the way, data typically passes through three programs: from the client browser, to the web server, to the CGI script, and back again to the browser. This is a natural model for the database access interaction we’re after— users can submit a database key to the server and receive the corresponding record as a reply page. We’ll go into CGI basics in depth later in this book, but as a first example, let’s start out with a simple interactive example that requests and then echoes back a user’s name in a web browser. The first page in this interaction is just an input form produced by the HTML file shown in Example 1-30. This HTML file is stored on the web server machine, and it is transferred to the web browser running on the client machine upon request. Example 1-30. PP4E\Preview\cgi101.html Interactive Page Notice how this HTML form names the script that will process its input on the server in its action attribute. This page is requested by submitting its URL (web address). When received by the web browser on the client, the input form that this code produces is shown in Figure 1-10 (in Internet Explorer here). Figure 1-10. cgi101.html input form page When this input form is submitted, a web server intercepts the request (more on the web server in a moment) and runs the Python CGI script in Example 1-31. Like the Step 6: Adding a Web Interface | 53 HTML file, this Python script resides on the same machine as the web server; it’s run on the server machine to handle the inputs and generate a reply to the browser on the client. It uses the cgi module to parse the form’s input and insert it into the HTML reply stream, properly escaped. The cgi module gives us a dictionary-like interface to form inputs sent by the browser, and the HTML code that this script prints winds up rendering the next page on the client’s browser. In the CGI world, the standard output stream is connected to the client through a socket. Example 1-31. PP4E\Preview\cgi-bin\cgi101.py #!/usr/bin/python import cgi form = cgi.FieldStorage() # parse form data print('Content-type: text/html\n') # hdr plus blank line print('Reply Page ') # html reply page if not 'user' in form: print('Who are you?
') else: print('Hello %s!
' % cgi.escape(form['user'].value)) And if all goes well, we receive the reply page shown in Figure 1-11—essentially, just an echo of the data we entered in the input page. The page in this figure is produced by the HTML printed by the Python CGI script running on the server. Along the way, the user’s name was transferred from a client to a server and back again—potentially across networks and miles. This isn’t much of a website, of course, but the basic principles here apply, whether you’re just echoing inputs or doing full-blown e-whatever. Figure 1-11. cgi101.py script reply page for input form If you have trouble getting this interaction to run on Unix-like systems, you may need to modify the path to your Python in the #! line at the top of the script file and make it executable with a chmod command, but this is dependent on your web server (again, more on the missing server piece in a moment). 54 | Chapter 1: A Sneak Preview Also note that the CGI script in Example 1-31 isn’t printing complete HTML: the and tags of the static HTML file in Example 1-30 are missing. Strictly speaking, such tags should be printed, but web browsers don’t mind the omissions, and this book’s goal is not to teach legalistic HTML; see other resources for more on HTML. GUIs versus the Web Before moving on, it’s worth taking a moment to compare this basic CGI example with the simple GUI of Example 1-28 and Figure 1-6. Here, we’re running scripts on a server to generate HTML that is rendered in a web browser. In the GUI, we make calls to build the display and respond to events within a single process and on a single machine. The GUI runs multiple layers of software, but not multiple programs. By contrast, the CGI approach is much more distributed—the server, the browser, and possibly the CGI script itself run as separate programs that usually communicate over a network. Because of such differences, the standalone GUI model may be simpler and more direct: there is no intermediate server, replies do not require invoking a new program, no HTML needs to be generated, and the full power of a GUI toolkit is at our disposal. On the other hand, a web-based interface can be viewed in any browser on any computer and only requires Python on the server machine. And just to muddle the waters further, a GUI can also employ Python’s standard library networking tools to fetch and display data from a remote server (that’s how web browsers do their work internally), and some newer frameworks such as Flex, Silverlight, and pyjamas provide toolkits that support more full-featured user interfaces within web pages on the client (the RIAs I mentioned earlier), albeit at some added cost in code complexity and software stack depth. We’ll revisit the trade-offs of the GUI and CGI schemes later in this book, because it’s a major design choice today. First, let’s preview a handful of pragmatic issues related to CGI work before we apply it to our people database. Running a Web Server Of course, to run CGI scripts at all, we need a web server that will serve up our HTML and launch our Python scripts on request. The server is a required mediator between the browser and the CGI script. If you don’t have an account on a machine that has such a server available, you’ll want to run one of your own. We could configure and run a full production-level web server such as the open source Apache system (which, by the way, can be tailored with Python-specific support by the mod_python extension). For this chapter, however, I instead wrote a simple web server in Python using the code in Example 1-32. We’ll revisit the tools used in this example later in this book. In short, because Python provides precoded support for various types of network servers, we can build a Step 6: Adding a Web Interface | 55 CGI-capable and portable HTTP web server in just 8 lines of code (and a whopping 16 if we include comments and blank lines). As we’ll see later in this book, it’s also easy to build proprietary network servers with low-level socket calls in Python, but the standard library provides canned implementations for many common server types, web based or otherwise. The socketserver module, for instance, supports threaded and forking versions of TCP and UDP servers. Third-party systems such as Twisted provide even more implementations. For serving up web content, the standard library modules used in Example 1-32 provide what we need. Example 1-32. PP4E\Preview\webserver.py """ Implement an HTTP web server in Python that knows how to run server-side CGI scripts coded in Python; serves files and scripts from current working dir; Python scripts must be stored in webdir\cgi-bin or webdir\htbin; """ import os, sys from http.server import HTTPServer, CGIHTTPRequestHandler webdir = '.' port = 80 # where your html files and cgi-bin script directory live # default http://localhost/, else use http://localhost:xxxx/ os.chdir(webdir) # run in HTML root dir srvraddr = ("", port) # my hostname, portnumber srvrobj = HTTPServer(srvraddr, CGIHTTPRequestHandler) srvrobj.serve_forever() # run as perpetual daemon The classes this script uses assume that the HTML files to be served up reside in the current working directory and that the CGI scripts to be run live in a cgi-bin or htbin subdirectory there. We’re using a cgi-bin subdirectory for scripts, as suggested by the filename of Example 1-31. Some web servers look at filename extensions to detect CGI scripts; our script uses this subdirectory-based scheme instead. To launch the server, simply run this script (in a console window, by an icon click, or otherwise); it runs perpetually, waiting for requests to be submitted from browsers and other clients. The server listens for requests on the machine on which it runs and on the standard HTTP port number 80. To use this script to serve up other websites, either launch it from the directory that contains your HTML files and a cgi-bin subdirectory that contains your CGI scripts, or change its webdir variable to reflect the site’s root directory (it will automatically change to that directory and serve files located there). But where in cyberspace do you actually run the server script? If you look closely enough, you’ll notice that the server name in the addresses of the prior section’s examples (near the top right of the browser after the http://) is always localhost. To keep this simple, I am running the web server on the same machine as the web browser; that’s what the server name “localhost” (and the equivalent IP address “127.0.0.1”) means. That is, the client and server machines are the same: the client (web browser) 56 | Chapter 1: A Sneak Preview and server (web server) are just different processes running at the same time on the same computer. Though not meant for enterprise-level work, this turns out to be a great way to test CGI scripts—you can develop them on the same machine without having to transfer code back to a remote server machine after each change. Simply run this script from the directory that contains both your HTML files and a cgi-bin subdirectory for scripts and then use http://localhost/… in your browser to access your HTML and script files. Here is the trace output the web server script produces in a Windows console window that is running on the same machine as the web browser and launched from the directory where the HTML files reside: ...\PP4E\Preview> python webserver.py mark-VAIO - - [28/Jan/2010 18:34:01] "GET /cgi101.html HTTP/1.1" 200 mark-VAIO - - [28/Jan/2010 18:34:12] "POST /cgi-bin/cgi101.py HTTP/1.1" 200 mark-VAIO - - [28/Jan/2010 18:34:12] command: C:\Python31\python.exe -u C:\Users \mark\Stuff\Books\4E\PP4E\dev\Examples\PP4E\Preview\cgi-bin\cgi101.py "" mark-VAIO - - [28/Jan/2010 18:34:13] CGI script exited OK mark-VAIO - - [28/Jan/2010 18:35:25] "GET /cgi-bin/cgi101.py?user=Sue+Smith HTTP /1.1" 200 mark-VAIO - - [28/Jan/2010 18:35:25] command: C:\Python31\python.exe -u C:\Users \mark\Stuff\Books\4E\PP4E\dev\Examples\PP4E\Preview\cgi-bin\cgi101.py mark-VAIO - - [28/Jan/2010 18:35:26] CGI script exited OK One pragmatic note here: you may need administrator privileges in order to run a server on the script’s default port 80 on some platforms: either find out how to run this way or try running on a different port. To run this server on a different port, change the port number in the script and name it explicitly in the URL (e.g., http://localhost:8888/). We’ll learn more about this convention later in this book. And to run this server on a remote computer, upload the HTML files and CGI scripts subdirectory to the remote computer, launch the server script on that machine, and replace “localhost” in the URLs with the domain name or IP address of your server machine (e.g., http://www.myserver.com/). When running the server remotely, all the interaction will be as shown here, but inputs and replies will be automatically shipped across network connections, not routed between programs running on the same computer. To delve further into the server classes our web server script employs, see their implementation in Python’s standard library (C:\Python31\Lib for Python 3.1); one of the major advantages of open source system like Python is that we can always look under the hood this way. In Chapter 15, we’ll expand Example 1-32 to allow the directory name and port numbers to be passed in on the command line. Using Query Strings and urllib In the basic CGI example shown earlier, we ran the Python script by filling out and submitting a form that contained the name of the script. Really, server-side CGI scripts can be invoked in a variety of ways—either by submitting an input form as shown so Step 6: Adding a Web Interface | 57 far or by sending the server an explicit URL (Internet address) string that contains inputs at the end. Such an explicit URL can be sent to a server either inside or outside of a browser; in a sense, it bypasses the traditional input form page. For instance, Figure 1-12 shows the reply generated by the server after typing a URL of the following form in the address field at the top of the web browser (+ means a space here): http://localhost/cgi-bin/cgi101.py?user=Sue+Smith Figure 1-12. cgi101.py reply to GET-style query parameters The inputs here, known as query parameters, show up at the end of the URL after the ?; they are not entered into a form’s input fields. Adding inputs to URLs is sometimes called a GET request. Our original input form uses the POST method, which instead ships inputs in a separate step. Luckily, Python CGI scripts don’t have to distinguish between the two; the cgi module’s input parser handles any data submission method differences for us. It’s even possible, and often useful, to submit URLs with inputs appended as query parameters completely outside any web browser. The Python urllib module package, for instance, allows us to read the reply generated by a server for any valid URL. In effect, it allows us to visit a web page or invoke a CGI script from within another script; your Python code, instead of a browser, acts as the web client. Here is this module in action, run from the interactive command line: >>> from urllib.request import urlopen >>> conn = urlopen('http://localhost/cgi-bin/cgi101.py?user=Sue+Smith') >>> reply = conn.read() >>> reply b'Reply Page \nHello Sue Smith!
\n' >>> urlopen('http://localhost/cgi-bin/cgi101.py').read() b'Reply Page \nWho are you?
\n' >>> urlopen('http://localhost/cgi-bin/cgi101.py?user=Bob').read() b'Reply Page \nHello Bob!
\n' 58 | Chapter 1: A Sneak Preview The urllib module package gives us a file-like interface to the server’s reply for a URL. Notice that the output we read from the server is raw HTML code (normally rendered by a browser). We can process this text with any of Python’s text-processing tools, including: • String methods to search and split • The re regular expression pattern-matching module • Full-blown HTML and XML parsing support in the standard library, including html.parser, as well as SAX-, DOM-, and ElementTree–style XML parsing tools. When combined with such tools, the urllib package is a natural for a variety of techniques—ad-hoc interactive testing of websites, custom client-side GUIs, “screen scraping” of web page content, and automated regression testing systems for remote server-side CGI scripts. Formatting Reply Text One last fine point: because CGI scripts use text to communicate with clients, they need to format their replies according to a set of rules. For instance, notice how Example 1-31 adds a blank line between the reply’s header and its HTML by printing an explicit newline (\n) in addition to the one print adds automatically; this is a required separator. Also note how the text inserted into the HTML reply is run through the cgi.escape (a.k.a. html.escape in Python 3.2; see the note under “Python HTML and URL Escape Tools” on page 1203) call, just in case the input includes a character that is special in HTML. For example, Figure 1-13 shows the reply we receive for form input Bob Smith—the in the middle becomes </i> in the reply, and so doesn’t interfere with real HTML code (use your browser’s view source option to see this for yourself); if not escaped, the rest of the name would not be italicized. Figure 1-13. Escaping HTML characters Step 6: Adding a Web Interface | 59 Escaping text like this isn’t always required, but it is a good rule of thumb when its content isn’t known; scripts that generate HTML have to respect its rules. As we’ll see later in this book, a related call, urllib.parse.quote, applies URL escaping rules to text. As we’ll also see, larger frameworks often handle text formatting tasks for us. A Web-Based Shelve Interface Now, to use the CGI techniques of the prior sections for our database application, we basically just need a bigger input and reply form. Figure 1-14 shows the form we’ll implement for accessing our database in a web browser. Figure 1-14. peoplecgi.html input page Coding the website To implement the interaction, we’ll code an initial HTML input form, as well as a Python CGI script for displaying fetch results and processing update requests. Example 1-33 shows the input form’s HTML code that builds the page in Figure 1-14. Example 1-33. PP4E\Preview\peoplecgi.htmlPeople Input Form To handle form (and other) requests, Example 1-34 implements a Python CGI script that fetches and updates our shelve’s records. It echoes back a page similar to that produced by Example 1-33, but with the form fields filled in from the attributes of actual class objects in the shelve database. As in the GUI, the same web page is used for both displaying results and inputting updates. Unlike the GUI, this script is run anew for each step of user interaction, and it reopens the database each time (the reply page’s action field provides a link back to the script for the next request). The basic CGI model provides no automatic memory from page to page, so we have to start from scratch each time. Example 1-34. PP4E\Preview\cgi-bin\peoplecgi.py """ Implement a web-based interface for viewing and updating class instances stored in a shelve; the shelve lives on server (same machine if localhost) """ import cgi, shelve, sys, os shelvename = 'class-shelve' fieldnames = ('name', 'age', 'job', 'pay') # cgi.test() dumps inputs # shelve files are in cwd form = cgi.FieldStorage() print('Content-type: text/html') sys.path.insert(0, os.getcwd()) # parse form data # hdr, blank line is in replyhtml # so this and pickler find person # main html template replyhtml = """People Input Form """ # insert html for data rows at $ROWS$ rowhtml = '%s \n' rowshtml = '' for fieldname in fieldnames: rowshtml += (rowhtml % ((fieldname,) * 3)) Step 6: Adding a Web Interface | 61 replyhtml = replyhtml.replace('$ROWS$', rowshtml) def htmlize(adict): new = adict.copy() for field in fieldnames: value = new[field] new[field] = cgi.escape(repr(value)) return new # values may have &, >, etc. # display as code: quoted # html-escape special chars def fetchRecord(db, form): try: key = form['key'].value record = db[key] fields = record.__dict__ # use attribute dict fields['key'] = key # to fill reply string except: fields = dict.fromkeys(fieldnames, '?') fields['key'] = 'Missing or invalid key!' return fields def updateRecord(db, form): if not 'key' in form: fields = dict.fromkeys(fieldnames, '?') fields['key'] = 'Missing key input!' else: key = form['key'].value if key in db: record = db[key] # update existing record else: from person import Person # make/store new one for key record = Person(name='?', age='?') # eval: strings must be quoted for field in fieldnames: setattr(record, field, eval(form[field].value)) db[key] = record fields = record.__dict__ fields['key'] = key return fields db = shelve.open(shelvename) action = form['action'].value if 'action' in form else None if action == 'Fetch': fields = fetchRecord(db, form) elif action == 'Update': fields = updateRecord(db, form) else: fields = dict.fromkeys(fieldnames, '?') # bad submit button value fields['key'] = 'Missing or invalid action!' db.close() print(replyhtml % htmlize(fields)) # fill reply from dict This is a fairly large script, because it has to handle user inputs, interface with the database, and generate HTML for the reply page. Its behavior is fairly straightforward, though, and similar to the GUI of the prior section. 62 | Chapter 1: A Sneak Preview Directories, string formatting, and security A few fine points before we move on. First of all, make sure the web server script we wrote earlier in Example 1-32 is running before you proceed; it’s going to catch our requests and route them to our script. Also notice how this script adds the current working directory (os.getcwd) to the sys.path module search path when it first starts. Barring a PYTHONPATH change, this is required to allow both the pickler and this script itself to import the person module one level up from the script. Because of the new way the web server runs CGI scripts in Python 3, the current working directory isn’t added to sys.path, even though the shelve’s files are located there correctly when opened. Such details can vary per server. The only other feat of semi-magic the CGI script relies on is using a record’s attribute dictionary (__dict__) as the source of values when applying HTML escapes to field values and string formatting to the HTML reply template string in the last line of the script. Recall that a %(key)code replacement target fetches a value by key from a dictionary: >>> D = {'say': 5, 'get': 'shrubbery'} >>> D['say'] 5 >>> S = '%(say)s => %(get)s' % D >>> S '5 => shrubbery' By using an object’s attribute dictionary, we can refer to attributes by name in the format string. In fact, part of the reply template is generated by code. If its structure is confusing, simply insert statements to print replyhtml and to call sys.exit, and run from a simple command line. This is how the table’s HTML in the middle of the reply is generated (slightly formatted here for readability): This text is then filled in with key values from the record’s attribute dictionary by string formatting at the end of the script. This is done after running the dictionary through a utility to convert its values to code text with repr and escape that text per HTML conventions with cgi.escape (again, the last step isn’t always required, but it’s generally a good practice). These HTML reply lines could have been hardcoded in the script, but generating them from a tuple of field names is a more general approach—we can add new fields in the future without having to update the HTML template each time. Python’s string processing tools make this a snap. Step 6: Adding a Web Interface | 63 In the interest of fairness, I should point out that Python’s newer str.format method could achieve much the same effect as the traditional % format expression used by this script, and it provides specific syntax for referencing object attributes which to some might seem more explicit than using __dict__ keys: >>> D = {'say': 5, 'get': 'shrubbery'} >>> '%(say)s => %(get)s' % D '5 => shrubbery' >>> '{say} => {get}'.format(**D) '5 => shrubbery' # expression: key reference # method: key reference >>> from person import Person >>> bob = Person('Bob', 35) >>> '%(name)s, %(age)s' % bob.__dict__ 'Bob, 35' >>> '{0.name} => {0.age}'.format(bob) 'Bob => 35' # expression: __dict__ keys # method: attribute syntax Because we need to escape attribute values first, though, the format method call’s attribute syntax can’t be used directly this way; the choice is really between both technique’s key reference syntax above. (At this writing, it’s not clear which formatting technique may come to dominate, so we take liberties with using either in this book; if one replaces the other altogether someday, you’ll want to go with the winner.) In the interest of security, I also need to remind you one last time that the eval call used in this script to convert inputs to Python objects is powerful, but not secure—it happily runs any Python code, which can perform any system modifications that the script’s process has permission to make. If you care, you’ll need to trust the input source, run in a restricted environment, or use more focused input converters like int and float. This is generally a larger concern in the Web world, where request strings might arrive from arbitrary sources. Since we’re all friends here, though, we’ll ignore the threat. Using the website Despite the extra complexities of servers, directories, and strings, using the web interface is as simple as using the GUI, and it has the added advantage of running on any machine with a browser and Web connection. To fetch a record, fill in the Key field and click Fetch; the script populates the page with field data grabbed from the corresponding class instance in the shelve, as illustrated in Figure 1-15 for key bob. Figure 1-15 shows what happens when the key comes from the posted form. As usual, you can also invoke the CGI script by instead passing inputs on a query string at the end of the URL; Figure 1-16 shows the reply we get when accessing a URL of the following form: http://localhost/cgi-bin/peoplecgi.py?action=Fetch&key=sue 64 | Chapter 1: A Sneak Preview Figure 1-15. peoplecgi.py reply page Figure 1-16. peoplecgi.py reply for query parameters As we’ve seen, such a URL can be submitted either within your browser or by scripts that use tools such as the urllib package. Again, replace “localhost” with your server’s domain name if you are running the script on a remote machine. To update a record, fetch it by key, enter new values in the field inputs, and click Update; the script will take the input fields and store them in the attributes of the class instance in the shelve. Figure 1-17 shows the reply we get after updating sue. Step 6: Adding a Web Interface | 65 Finally, adding a record works the same as in the GUI: fill in a new key and field values and click Update; the CGI script creates a new class instance, fills out its attributes, and stores it in the shelve under the new key. There really is a class object behind the web page here, but we don’t have to deal with the logic used to generate it. Figure 1-18 shows a record added to the database in this way. Figure 1-17. peoplecgi.py update reply Figure 1-18. peoplecgi.py after adding a new record In principle, we could also update and add records by submitting a URL—either from a browser or from a script—such as: http://localhost/cgi-bin/ peoplecgi.py?action=Update&key=sue&pay=50000&name=Sue+Smith& ...more... 66 | Chapter 1: A Sneak Preview Except for automated tools, though, typing such a long URL will be noticeably more difficult than filling out the input page. Here is part of the reply page generated for the “guido” record’s display of Figure 1-18 (use your browser’s “view page source” option to see this for yourself). Note how the < and > characters are translated to HTML escapes with cgi.escape before being inserted into the reply:
key name age job pay key name age job pay As usual, the standard library urllib module package comes in handy for testing our CGI script; the output we get back is raw HTML, but we can parse it with other standard library tools and use it as the basis of a server-side script regression testing system run on any Internet-capable machine. We might even parse the server’s reply fetched this way and display its data in a client-side GUI coded with tkinter; GUIs and web pages are not mutually exclusive techniques. The last test in the following interaction shows a portion of the error message page’s HTML that is produced when the action is missing or invalid in the inputs, with line breaks added for readability: >>> from urllib.request import urlopen >>> url = 'http://localhost/cgi-bin/peoplecgi.py?action=Fetch&key=sue' >>> urlopen(url).read() b'\n People Input Form \n\n