Oracle Enterprise Manager 10g Grid Control Implementation Guide (Osborne Press Series) Mc Graw.Hill.Osborne.Media.Oracle.Enterprise.Manager.10g.Grid.Control.Implementation.Guide.Nov.200
User Manual: Pdf
Open the PDF directly: View PDF 
.
Page Count: 769
| Download | |
| Open PDF In Browser | View PDF |
® Oracle Enterprise Manager 10g Grid Control Implementation Guide Michael New New York Chicago San Francisco Lisbon London Madrid Mexico City Milan New Delhi San Juan Seoul Singapore Sydney Toronto Copyright © 2009 by The McGraw-Hill Companies, Inc. All rights reserved. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher. ISBN: 978-0-07-159503-2 MHID: 0-07-159503-1 The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-149275-1, MHID: 0-07-149275-5. All trademarks are trademarks of their respective owners. Rather than put a trademark symbol after every occurrence of a trademarked name, we use names in an editorial fashion only, and to the benefit of the trademark owner, with no intention of infringement of the trademark. Where such designations appear in this book, they have been printed with initial caps. McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales promotions, or for use in corporate training programs. To contact a representative please visit the Contact Us page at www.mhprofessional.com. Information has been obtained by Publisher from sources believed to be reliable. However, because of the possibility of human or mechanical error by our sources, Publisher, or others, Publisher does not guarantee to the accuracy, adequacy, or completeness of anyinformation included in this work and is not responsible for any errors or omissions or the results obtained from the use of such information. Oracle Corporation does not make any representations or warranties as to the accuracy, adequacy, or completeness of any information contained in this Work, and is not responsible for any errors or omissions. TERMS OF USE This is a copyrighted work and The McGraw-Hill Companies, Inc. (“McGraw-Hill”) and its licensors reserve all rights in and to the work. Use of this work is subject to these terms. Except as permitted under the Copyright Act of 1976 and the right to store and retrieve one copy of the work, you may not decompile, disassemble, reverse engineer, reproduce, modify, create derivative works based upon, transmit, distribute, disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hill’s prior consent. You may use the work for your own noncommercial and personal use; any other use of the work is strictly prohibited. Your right to use the work may be terminated if you fail to comply with these terms. THE WORK IS PROVIDED “AS IS.” McGRAW-HILL AND ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS TO THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR OTHERWISE, AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. McGraw-Hill and its licensors do not warrant or guarantee that the functions contained in the work will meet your requirements or that its operation will be uninterrupted or error free. Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for any inaccuracy, error or omission, regardless of cause, in the work or for any damages resulting therefrom. McGraw-Hill has no responsibility for the content of any information accessed through the work. Under no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, incidental, special, punitive, consequential or similar damages that result from the use of or inability to use the work, even if any of them has been advised of the possibility of such damages. This limitation of liability shall apply to any claim or cause whatsoever whether such claim or cause arises in contract, tort or otherwise. I dedicate this book to my daughter, Leah, and to my late brother, Robbie. About the Author Michael New is a senior information systems architect with 17 years of experience in the software industry. After receiving his B.S. from M.I.T. in Aeronautics and Astronautics, he worked for six years as a software engineer, principally on the GPS satellite program at Rockwell. For the past 11 years, Michael has worked as an Oracle core and Oracle Applications DBA. He began as a full-time DBA, moved into consulting with an Oracle partner, then served as a Technical Manager for Oracle Consulting before launching his own firm, New DB Solutions. Michael has implemented the full array of Oracle products for clients in many industries, including the Oracle E-Business Suite, RAC, Data Guard, RMAN, Application Server, and Grid Control (in which he is a recognized authority). Michael has authored several white papers on database administration, has been a presenter at Oracle OpenWorld, and has been a technical editor for a McGraw-Hill Professional book on Oracle 10g. Michael lives in New York City with his five-year-old daughter, Leah. You can reach him at Michael@ NewDBsolutions.com or through his web site, www.newdbsolutions.com. About the Technical Editors Phil Choi worked at Oracle for several years documenting the Enterprise Manager suite of products. Before Oracle, he worked in a variety of industries, including consulting, software, and e-commerce. Phil has yet to put his Mechanical Engineering degree to good use, but the degree in Creative Writing has come in handy during his attempts to write short stories and essays for publication. Phil (and his dog Ajax) lives and works in San Francisco. Edward Whalen is the founder and Chief Technology Officer of Performance Tuning Corporation. During his career, he completed many complex consulting projects and continues to write books and technical papers on various Oracle and Microsoft technologies. Ed has published more than eight books on SQL Server and Oracle, most recently Microsoft SQL Server 2005 Administrator’s Companion and Oracle Database 10g Linux Administration. Ed is recognized as a leader in database design and performance optimization and provides critical services to companies worldwide. Contents Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii xv part I Install Grid Control 1 2 Overview of the Grid Control Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 What Is Grid Computing? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . What Is Grid Control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Management Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Management Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Management Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Flow Between Grid Control Components . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control, Database Control, and AS Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control vs. Database Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control vs. AS Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agents for Grid Control, Database Control, and AS Control . . . . . . . . . . . . . . . Grid Control, Database Control, and AS Control: All Together Now . . . . . . . . . 4 7 10 13 14 15 17 19 22 23 24 26 26 Grid Control Preinstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Key Architectural Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Decide How Many Grid Control Environments to Build . . . . . . . . . . . . . . . . . . How Many Regional Sites Are Required? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Make Required Installation Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Configuration Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Host Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fully Qualify Host Name References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Host Name Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use Static IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Connectivity Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 35 36 41 47 49 49 52 53 53 v vi Oracle Enterprise Manager 10g Grid Control Implementation Guide 3 Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware Installation Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hardware Operating Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Certification Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Required OS Groups and Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Required Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stop Database Listeners Using Port 1521 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Synchronize OS Timestamps/Time Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Confirm Platform-Specific Software Requirements . . . . . . . . . . . . . . . . . . . . . . 55 56 57 60 60 61 65 70 70 72 Grid Control Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Gather Needed Installation Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Address Installation Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Initialize the Oracle User Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up X Server (UNIX Only) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set/Unset OS Environment Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Running the Installer with Desired Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Silent Installation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Static Ports Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Starting and Monitoring the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Grid Control Using a New Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Installation Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Installation Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Language Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Inventory Directory and Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . Product-Specific Prerequisite Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Optional Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specify Security Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Configuration Manager Registration . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration Assistants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . End of Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Grid Control Using an Existing Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steps to Install the OMR Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run the Installer—GC Existing Database Option . . . . . . . . . . . . . . . . . . . . . . . Install Standalone Agents on Dedicated OMR Nodes . . . . . . . . . . . . . . . . . . . . Log in to the Web Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install the Grid Control Security Certificate in Your Browser . . . . . . . . . . . . . . . Install an Additional OMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 77 83 83 85 87 87 88 89 91 91 91 93 93 94 95 95 97 99 100 101 101 103 103 105 111 114 114 115 116 Contents 4 Grid Control Post Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Patch Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apply the Latest Grid Control Patch Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apply Latest Database Patch Set Certified for OMR . . . . . . . . . . . . . . . . . . . . . Apply the Latest EM Critical Patch Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apply Required Grid Control One-off Patches . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Management Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reduce Grid Control Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Oracle User Environment on OMS Hosts . . . . . . . . . . . . . . . . . . . . . . . Modify the Default Console Timeout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure OMS for Failover and Load Balancing . . . . . . . . . . . . . . . . . . . . . . . Tune OMS Thread Pool Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Add OMR Alias to OMS tnsnames.ora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Back Up Critical OMS Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oracle Management Repository Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Oracle User Environment on OMR Nodes . . . . . . . . . . . . . . . . . . . . . . Confirm That Listeners Load-Balance Across RAC Nodes . . . . . . . . . . . . . . . . . Configure i SQL*Plus Access in Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . Modify the Data Retention Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modify Job Purge Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Secure the emkey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Grid Control Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure and Tune the OMR Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Oracle Configuration Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 vii Preinstall Standalone Management Agents 120 121 123 124 125 129 130 135 138 138 139 140 140 141 141 141 142 148 150 150 151 152 152 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 An Overview of Common Agent Preinstallation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Key Installation Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Select Agent Installation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use an Existing User or Create a Separate Agent User . . . . . . . . . . . . . . . . . . . Install a Local or Cluster Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure a Server Load Balancer First . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Use an LDAP or Local Agent User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Choose to Secure or Not Secure the Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Disk Space Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent CPU Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent RAM Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary of Agent Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Initialize the oracle User Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Confirm No Existing Agent Is Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Change Response File if Using an SLB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prepare for Cross-Platform Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . Prepare Targets for Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gather Needed Agent Installation Information . . . . . . . . . . . . . . . . . . . . . . . . . 157 158 158 162 163 165 165 166 166 166 167 168 168 168 168 169 170 171 173 173 viii Oracle Enterprise Manager 10g Grid Control Implementation Guide 6 Install Management Agents via Agent Deploy . . . . . . . . . . . . . . . . . . . . . . . . . 175 Agent Deploy Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Required Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure SSH User Equivalence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up SSH Server (SSHD) on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Back Up Current SSH Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set ORACLE HOME on OMS Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run sshConnectivity.sh Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify SSH User Equivalence Is Configured . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Time Zone for SSH Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Validate All Command Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modify Agent Deploy Properties File for SLB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Choose Inventory Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Agent User Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prepare for Cross-Platform Agent Push . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Include Additional Files (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run Agent Deploy Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fresh Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NFS Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Upgrade Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Deploy Post-Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Non-NFS Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NFS Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Troubleshoot Agent Deploy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Install Management Agents Locally 176 178 178 179 179 179 180 184 184 185 187 187 188 189 189 190 192 195 196 197 198 198 198 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 nfsagentinstall Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Master Agent on Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install NFS Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . agentDownload Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prepare for agentDownload Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Copy agentDownload Script from OMS to Target Host . . . . . . . . . . . . . . . . . . . Execute agentDownload on Target Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Cloning Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Silent Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interactive Agent Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Required Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run the Interactive Installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Post-Installation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Agent User Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Confirm Agent Is Working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Refresh Host Configuration (If Needed) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run agentca for Cluster Agent (Windows Only) . . . . . . . . . . . . . . . . . . . . . . . . Confirm Agent Restart on Reboot Is Configured . . . . . . . . . . . . . . . . . . . . . . . . Back Up the Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 205 209 211 213 214 217 219 223 225 228 228 229 236 237 238 240 240 240 241 Contents 8 Install Grid Control Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Install and Configure the EM Java Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Determine Whether You Need the EM Java Console . . . . . . . . . . . . . . . . . . . . Installation Steps for the EM Java Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Start the EM Java Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure Change Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Adobe SVG Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install EM Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install Oracle Configuration Manager Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 ix EM Login and Component Control 244 245 246 249 250 262 263 264 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Log in to EM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Web Console Login to Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EM Java Console Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AS Console Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iSQL*Plus Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metrics Browser Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control EM Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control iSQL*Plus Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control the Management Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control the AS Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control the Management Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control the Management Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Windows EM Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EM Startup/Shutdown Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 269 272 273 274 276 278 278 279 279 280 286 290 292 part II Configure and Maintain Grid Control 10 General Console Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tour of the Setup and Preferences Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Follow an Event from Trigger to Notification . . . . . . . . . . . . . . . . . . . . . . . . . . Reasoning Behind this Chapter’s Menu Navigation Order . . . . . . . . . . . . . . . . Set Up Administrators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Administrators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Administrator Preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Preferred Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Define Notification Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Notification Schedules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create and Subscribe to Notification Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enable Patching Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Complete the Patching Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Schedule the RefreshFromMetalink Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 299 303 305 307 307 313 317 320 330 330 334 339 356 356 362 x Oracle Enterprise Manager 10g Grid Control Implementation Guide 11 Configure Target Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 Remove Targets from Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ensure Agents Are Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Back Up targets.xml File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Confirm No Targets Are in Blackout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Remove Targets from Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discover and Configure Targets for Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Three Paths to Discover/Configure a Target . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ASM Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Database Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Listener Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Host Additional Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OracleAS Discovery and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enter Target Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grant Management Pack Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Management Options: Packs, Plug-ins, and Connectors . . . . . . . . . . . . . . . . . . Schedule Blackouts for Planned Downtimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Blackouts in the Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Blackouts at the Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Configure Group and Service Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 Group Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create a Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Group Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Service Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SLA Objectives for Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Elements of Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure a Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Tune Metrics and Policies 367 367 367 367 368 370 371 376 383 386 407 409 411 413 414 414 421 421 427 430 431 435 438 439 441 445 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489 Change Default Metrics and Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Change Default Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Change Default Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enable Metric Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metric Baseline Concepts and Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Metric Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disabling Metric Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Add Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Add Corrective Actions to Metrics or Policies . . . . . . . . . . . . . . . . . . . . . . . . . . Implement User-Defined Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create an OS User-Defined Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create a SQL User-Defined Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 492 501 504 506 509 514 515 516 518 521 522 527 14 Contents xi Use Metric Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Metric Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Apply Metric Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leverage Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Create Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maintain Custom Metrics when Applying Monitoring Templates . . . . . . . . . . . Apply Monitoring Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compare Settings Between Targets and the Template . . . . . . . . . . . . . . . . . . . . 529 530 532 534 536 540 541 544 Backup and Recovery of Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 Introduction to Grid Control B&R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control B&R Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resetting Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recommended Grid Control Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OMR Database Backup and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Back Up the OMR Database in Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . Recover the OMR Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leverage Flashback Technology for OMR Database Recovery . . . . . . . . . . . . . Manage OMR Database Backups in Grid Control . . . . . . . . . . . . . . . . . . . . . . . Grid Control Software Backup and Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OMR Database Software Backup and Restoration . . . . . . . . . . . . . . . . . . . . . . . OMS Software Backup and Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent Software Backup and Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Configure Grid Control for High Availability and Disaster Recovery . . . . . . . . 623 Grid Control High Availability Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . OMS HA Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OMR Database HA Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent HA Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grid Control Disaster Recovery Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . OMR Database DR Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . OMS DR Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agent DR Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Securing Grid Control Data Transfer 551 551 554 557 560 561 597 606 608 612 613 615 619 625 627 633 636 637 639 644 648 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 651 Enable EM Framework Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enable Framework Security Between Agents and OMS . . . . . . . . . . . . . . . . . . . Secure Repository Data Transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Secure Console Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Enforce HTTPS Between the Browser and AS Console . . . . . . . . . . . . . . . . . . . Enforce HTTPS Between Browser and OHS . . . . . . . . . . . . . . . . . . . . . . . . . . . Security Considerations for Console Web Cache Access . . . . . . . . . . . . . . . . . . Limit GC Console Access to Certain Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . 656 656 663 675 676 677 679 680 xii Oracle Enterprise Manager 10g Grid Control Implementation Guide 17 Maintain and Tune Grid Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685 Perform Routine Grid Control Maintenance Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Weekly Online Maintenance Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monthly Offline Maintenance Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gather and Document Grid Control Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Types of Grid Control Metrics to Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Specific Grid Control Metrics to Gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Procedure to Evaluate Grid Control Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . Tune Grid Control to Reduce Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reduce High CPU Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resolve Loader Backlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minimize Rollup Delays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluate Job, Notification, and Alert Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . Tune I/O Bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Improve Poor Console Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Platform-Specific Tuning Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . 686 687 693 699 699 703 708 709 710 715 719 722 723 727 734 Appendixes Online at www.OraclePressBooks.com A. Agent Subsystems B. Certification and Platform-Specific Installation Requirements C. Configure and Tune a Database for the Management Repository D. Set Up SSH User Equivalence E. Configure Grid Control Automatic Startup F. RMAN Scripts from Oracle-Suggested Backup Strategy in Grid Control G. Configuring OAS Using Oracle Net Manager H. Tools for Grid Control Performance Metrics I. How to Install the Grid Control Security Certificate in Your Browser Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737 Acknowledgments This book sprung from material for a Grid Control “Deep Dive” class that I developed and taught to Oracle Consulting personnel at the behest of Dennis Horton, a Senior Director at Oracle. Without his impetus, I would never have tackled this book. Thanks to the Oracle Press team at McGraw-Hill Professional: Lisa McClain, Mandy Canales, Jennifer Housh; the copyedit team, notably Vastavikta Sharma, Carolyn Welch, and Bill McManus; and the Illustration and Production departments for their work in producing this book. I am very grateful to Ed Whalen and Phil Choi for their roles as technical reviewers. Ed introduced me to the McGraw-Hill team by bringing me in as a technical editor for his book, Oracle Database 10g Linux Administration (Oracle Press, McGraw-Hill Professional, 2005). Later, he told me about McGraw-Hill’s desire to publish a Grid Control book. After I signed up to write this book, Ed provided his company’s computer resources as a platform for most of the Grid Control installation and configuration steps shown in the book. I am also largely indebted to Phil Choi, formerly the lead editor at Oracle for Enterprise Manager products, for extensively editing the book’s first draft. Phil taught me a great deal about technical writing. I greatly appreciate Oracle’s help in supplying and validating technical content as needed. Thanks to Oracle Support in answering some tough questions along the way, particularly Mathieu Hornsperger for explaining the inner workings of Grid Control security. Narain Jagathesan and Anirban Chatterjee in the Enterprise Manager development group at Oracle provided much of the detailed information about the Management Agent given in Appendix A (available online at www.OraclePressBooks.com). xiii xiv Oracle Enterprise Manager 10g Grid Control Implementation Guide I’d also like to acknowledge my Oracle mentors: Howard Ostrow, a great all-around technical resource and RAC expert; Matthew Burke, a DBA master who tutored me for four years in DBA paths less traveled and encouraged me to present my first white paper; and Ashok Nagabothu, who took me, when a fledgling DBA, under his wing. I am grateful to my family and friends for seeing me through a trying time personally while writing this book. My brother Jon, his wife Gina, their son Daniel, my mother, and good friends Adam Bierman and Brad Stonberg were very supportive. I am indebted to Amy Seawright, erstwhile companion, who brought me back to life before my mistress, Oracle, reclaimed it. Finally, I thank my daughter, Leah, and my muse for resurrecting me yet again, once and for all. Introduction This is the first book dedicated solely to the subject of implementing Oracle Enterprise Manager 10g Grid Control. It was only a matter of time before such a book was published. With the first release (10.1) of Grid Control roughly seven years ago, Oracle ushered in a true system management product. The latest release (10.2) of the product is enjoying large-scale adoption by companies looking to centrally manage their complete IT infrastructure—not just their Oracle systems, but non-Oracle ones as well. Firms around the world are significantly cutting IT expenditures by administering their data center components with Grid Control. As you would expect from such a capable software product, the Enterprise Manager 10g Grid Control documentation library from Oracle Corporation is bulky, yet still does not contain solutions to certain common installation and configuration hurdles. Answers are found in supplemental material available on OracleMetaLink, the Oracle Technology Network (OTN), and elsewhere on the Web. (How much third-party material alone is out there? Google “Oracle Enterprise Manager 10g Grid Control” and you’ll get 122,000 hits.) All in all, it’s a lot of material to digest, especially if you’re new to Grid Control. Contrast a Grid Control novice’s first installation using product and supplemental documentation with that of an expert who has performed many installations, having already waded through and assimilated the reference material and learned the bugs, workarounds, optimal installation order, shortcuts, tips, and tricks. The expert will likely implement Grid Control successfully, on time and under budget, whereas the novice may not be so fortunate. The aim of this book is to lend an expert hand to a DBA who is not intimately acquainted with Grid Control so that he or she can implement it according to best practices, as quickly and painlessly as possible. This book distills what I’ve learned from completing many Grid Control engagements over the past seven years. Each implementation was unique: many were large installations at Fortune 500 companies, while others were smaller, proof-of-concept assignments; the platforms were different, as were the high availability requirements. Yet, these engagements also shared many commonalities: I ran across same issues over and over again—some xv xvi Oracle Enterprise Manager 10g Grid Control Implementation Guide generic, some platform-specific—and found myself returning to the same reference material for tried-and-true solutions. I documented the process for customers and gathered Grid Control collateral from colleagues. When I was with Oracle Consulting, I funneled this knowledge and experience into a week-long Grid Control “Deep Dive,” a hands-on laboratory course that I developed and taught to Oracle consultants. More than anything else, it was this trial by fire in teaching fellow consultants—the toughest students—that motivated me to want to share my methods with a wider audience. This book is a “best-of” compilation of all these Grid Control experiences, a how-to manual on which novice and aspiring expert can both rely. I provide step-by-step instructions on how to install, configure, maintain, and tune Grid Control releases 10.2.0.3 and 10.2.0.4 for all operating systems on which it is certified. The book does not cover how to upgrade an existing OEM 9.2 or 10.1 installation to the latest 10.2 version. I concentrate on how to implement Grid Control 10.2, not how to use it ad hoc (such as to tune managed targets). While configuring Grid Control is the book’s central focus, I use Grid Control to configure itself whenever possible to demonstrate its features. By turning Grid Control on itself, so to speak, you will learn how to use it. For example, the Chapter 14 material on backup and recovery of the Oracle Management Repository Database applies equally to target databases. The appendices for the book are available at www.OraclePressBooks.com on either the Downloads page or the book product page. The material there lends itself better to soft copy because you can copy the provided commands, scripts, etc. for setting up Grid Control to your clipboard. This book is intended primarily for Oracle DBAs with at least a few years of experience who want to install and configure Grid Control to administer their IT infrastructure, as well as for system and network administrators assisting in the process. In addition, the book is suitable for IT managers responsible for a Grid Control project or just interested in gaining insight into the product’s capabilities to administer both Oracle and non-Oracle products in their data centers. To effectively capitalize on the book, those in management roles should have at least a basic knowledge of Oracle technologies. Conventions Throughout the book, I refer to two main operating systems: Windows and UNIX. The term “UNIX” refers to all variants of that OS, including Linux. UNIX syntax is used unless the equivalent Windows syntax widely differs. The most prevalent difference is between the UNIX forward slash (/) and the Windows backslash (\) in directory specifications. Following are the conventions used throughout the book: Term MeaningText in italicized angle brackets represents a variable. Substitute a value for the variable text and do not type the angle brackets. [ parameters ] Text in square brackets represents one or more optional parameters for commands. Enter any optional parameters and do not type the square brackets. \ In UNIX, the backslash continuation character at the end of a line permits entering a single command on multiple lines. Alternatively, on any platform, you can omit the backslash and type the entire command on a single line, provided it adheres to OS-specific restrictions on the maximum number of characters allowed per line. In this book, a backslash at the end of a line signifies to choose one of these methods to enter the command. Part I Install Grid Control This page intentionally left blank Chapter 1 Overview of the Grid Control Architecture 3 4 Oracle Enterprise Manager 10g Grid Control Implementation Guide I f this book were a novel, the first sentence would aspire to emulate the masters. Something along the lines of “All Children, except one, grow up.”1; “It was the best of times, it was the worst of times…”2; or “As I walked through the wilderness of this world…”3. But this isn’t a novel. So let me begin more humbly. This chapter takes a high-level look at the architecture of Oracle Enterprise Manager 10g Grid Control (referred to throughout the book as Grid Control or GC), and includes descriptions of its core components. We’ll “kick the tires,” so to speak, then open the hood of the Grid Control engine, stick our heads in, and take a closer look at its parts and inner workings. I’m not asking you to remove these parts, spread them across the floor of your garage, and disassemble them one by one. However, near the end of the chapter, I will trace how data flows between all parts of the GC engine. Tip Most of this chapter deals with GC architecture, delving deeper into the topic than you might expect for a first chapter. My rationale is to establish a solid foundation you can refer back to as you progress through the book. If you’re in a rush, skim the chapter and move on to Chapter 2, which covers the pre installation process. This chapter answers the following questions: ■■ What is grid computing? ■■ What is Grid Control? ■■ What are the main GC components? ■■ What are the subcomponents that make up each component, and what are their functions? ■■ What protocols do components use to communicate with one another? ■■ What wiring enables communication between components for Console requests, Agent uploads, and alert notifications? ■■ What is the difference between Grid Control, Database Control, and Application Server Control? What Is Grid Computing? Before we begin an in-depth discussion of Grid Control architecture, we need to step back and ask this question: What is grid computing? Any description of Grid Control would be remiss without mentioning its namesake, “the grid.” Introduced in the late 1990s, the grid refers to grid 1 Peter Pan by J.M. Barrie, 1911. A Tale of Two Cities by Charles Dickens, 1859. 3 The Pilgrim’s Progress by John Bunyan, 1678. 2 Chapter 1: Overview of the Grid Control Architecture 5 computing, the next generation of computing that allows you to virtualize IT resources, including data, processing, servers, storage capacity, databases, and network bandwidth, as though they were utilities. The term was first coined to equate an IT department’s delivery of services with an energy company’s grid that supplies electrical power to its customers. In such a “utility computing” environment, applications rely on a grid infrastructure just as appliances rely on electricity from outlets. You don’t know where the generators are or how the electric company does its wiring. You just want to plug in an appliance and get electricity. The same applies to an application—you want it to work just like a utility. One of the most salient examples of grid computing is the Search for Extra-Terrestrial Intelligence At Home (SETI@home), a computing project that harnesses the processing power of over five million idle, Internet-connected computers to analyze signal data from radio telescopes in an attempt to detect intelligent life beyond Earth. Grid computing is based on an open set of standards and protocols, such as the Open Grid Services Architecture (OGSA), which describes a Service Oriented Architecture (SOA), enabling communication across heterogeneous, geographically dispersed sites. With grid computing, organizations optimize data and computing resources by pooling and sharing them across networks. Grid computing is another step in the evolution of IT developments, such as the Web, peer-to-peer computing, distributed computing, and virtualization technologies,4 but it is distinguished from these technologies in the following ways: ■■ Like the Web, grid computing conceals complexity, but unlike the Web, which mainly enables communication, grid computing enables collaboration to achieve common business goals. ■■ Like peer-to-peer computing, grid computing allows individual users to share files, but grid computing also allows many-to-many sharing—not only file sharing but all computational resources as well. ■■ Like distributed computing, grids bring computing resources together, but grids can be geographically distributed and heterogeneous. ■■ Like virtualization technologies, grid computing virtualizes IT resources but on a much larger scale in terms of computing power and storage. A grid strategy provides the following features: ■■ Scalable, highly available database clustering ■■ Load balancing ■■ Automatic self-tuning and self-management ■■ Storage virtualization ■■ Resource virtualization ■■ Grid management software 4 MCADCafe, April 5, 2004, Jeff Rowe. Oracle TIGHT / Oracle Enterprise Manager 10g Grid Control Implementation Guide / Michael New / 149275-5 6 Oracle Enterprise Manager 10g Grid Control Implementation Guide Clients Tier 1—Client WAN TrafficMgrs (load balancers) Routers Switches Tier 2—Application Server OracleAS Cluster .NET ODBC Oracle Net OCI JDBC OTHER Switches Routers RAC Interconnect RAC Database Cluster Note: WAN traffic managers, routers, and switches are redundant FIGURE 1-1. Tier 3—Database ASM Disk Group ASM Disk Group (DATA) (FLASH) Database Storage Subsystem Operating System A typical large grid computing infrastructure Figure 1-1 illustrates a typical real-world large grid computing architecture using Oracle technology. As shown in the figure, enterprise grid computing provides highly available database services to applications by means of a flexible component provisioning (i.e., deployment) topology. This topology integrates all IT resources—servers, storage, databases, applications, and networks—to provide provisioning on demand. These resources can be divided into three tiers. Let’s cover each tier from the user end to the back end. ■■ Tier 1—Client The client tier is the entry point for all application client connections. These are the client PCs, PDAs, ATMs, and other acronyms all requesting something that only IT can deliver—data. ch01.indd 6 11/1/08 12:23:35 PM Chapter 1: Overview of the Grid Control Architecture 7 ■■ Tier 2—Application Server The Application Server tier consists of application servers and web servers, which Oracle provides in Oracle Application Server (OracleAS) 10g. Figure 1-1 depicts a three-node OracleAS Cluster, with each node containing an application server and a web server. This cluster communicates with the database cluster via various protocols and specifications, including Open Database Connectivity (ODBC), Java Database Connectivity (JDBC), and Oracle Call Interface (OCI). ■■ Tier 3—Database The database tier is comprised of two main parts: ■■ Database cluster(s) The emergence of database clustering is made possible economically by the recent introduction of small multiprocessor hosts, low-cost blade servers, and the open source Linux operating system. Each Real Application Clusters (RAC) cluster presents a single database spanning all nodes in that cluster. In Figure 1-1, the cluster is composed of a four-node RAC database. (RAC nodes are shown in groups of two because you typically add two nodes at a time when more database server resources are required.) RAC offers the flexibility to add or remove nodes as capacity demands fluctuate. RAC also provides hardware failure protection; it can fence off failed nodes, redistribute processing evenly across the surviving nodes, and reconnect clients transparently to the remaining nodes.5 ■■ Database storage subsystem The storage subsystem is an independent collection of low-cost disk devices. The example in Figure 1-1 uses the Oracle solution of Automatic Storage Management (ASM) disk groups. Each disk group (two in this case, named DATA and FLASH) is composed of multiple disk devices (three in this example). ASM partitions the disk space and evenly distributes the data storage throughout the storage array. If you add or remove storage arrays, ASM “automagically”6 redistributes the data storage. On all tiers, the switches, routers, and WAN traffic managers (which also provide load balancing) are redundant. Figure 1-1 does not include a disaster recovery (DR) site, but such a site is essential to any highly available architecture. The DR site would be similarly configured, although perhaps scaled down. Many books have been written about grid computing, and this cursory discussion on the subject certainly cannot do it justice. It is simply intended to introduce you to the concept of the grid architecture so that you can place Grid Control in its proper context. I should note that, while a grid strategy is undoubtedly the right technological direction for companies to gravitate towards, your firm doesn’t need to be fully or even partially exploiting grid technology to take advantage of Grid Control. What Is Grid Control? With its usual uncanny foresight into the direction of IT technology, Oracle changed the name of Oracle Enterprise Manager 9i to Oracle Enterprise Manager 10g Grid Control when this new version of the product was first introduced in 2003. At that time, most corporate IT professionals were still relatively unversed in grid computing concepts, and their IT sites were anything but Grid compliant—most sites still aren’t, though the trend of computing as a commodity has 5 Transparent Application Failover (TAF), an Oracle database feature, only functions when applications are enabled for it. 6 My mentor and friend, Howard Ostrow, a RAC expert for Oracle Consulting Services, is fond of this phrase in describing ASM’s amazing capabilities. 8 Oracle Enterprise Manager 10g Grid Control Implementation Guide certainly caught on. To address the challenges of companies whose business needs change faster than their IT departments can adapt, Oracle introduced the three products we’ve spoken of to enable an Oracle grid solution: Oracle Application Server 10g (Tier 2), Oracle Database 10g and the accompanying ASM solution (Tier 3), and Grid Control to manage the entire lot. Given that these three products together are integral to implementing a grid strategy, it should come as no surprise that the “g” in “10g” in each of the product names stands for “grid.” Grid Control manages the grid and, as you will see in this chapter, is architected itself in a “grid-compliant” manner; that is, it is built on the same three tiers and, yes, it even monitors itself. Grid Control is an Oracle Web application that centralizes management of your enterprise IT grid (or even non grid) environment. Grid Control allows administrators to monitor the entire technology stack in the grid, including both Oracle and non-Oracle components. This flagship systems management product can manage over 100 types of Oracle and non-Oracle targets (system components) configured in any architecture. These components include routers, load balancers, switches, business applications, application servers, databases, networks, storage, and hosts, most of which inhabit the example in Figure 1-1. If a component within the grid goes offline or begins to experience performance issues, Grid Control automatically triggers an alert to notify an administrator so that he or she can take appropriate action. Grid Control is also capable of automatically replying to alerts with pre defined “fixit” jobs. Here are just some of Grid Control’s many features and how you can use them: ■■ Administer the database server tier (Tier 3 in Figure 1-1) using RAC management ■■ Monitor Automatic Storage Management (ASM) instances, Oracle’s solution to a shared array of disks in a database storage subsystem (Tier 3 in Figure 1-1) ■■ Manage application and web servers (Tier 2 in Figure 1-1) as part of providing service level management, which includes setting business service goals for transaction performance and business processes, monitoring and tuning service performance and usage, diagnosing root causes of problems, and reporting ■■ Monitor non-Oracle components such as WAN traffic managers (in Tier 1)—for example, the F5 BigIP Local Traffic Manager ■■ View overall system and service status at a glance using the home page and dashboards akin to monitors at a hosting site that allow drill-down to analyze the root cause of a problem ■■ Add alerts, security policies, and thresholds for all component types, and set rules for alert notifications ■■ Manage many databases, hosts, and other components as one using templates ■■ Bare metal provision new operating systems using pre tested software image libraries, and deploy new databases and application servers ■■ Manage the life cycle of the Oracle and OS patch process with direct connections to OracleMetaLink, proactive patch notifications, and automated distribution ■■ Automate tasks at the operating system and database levels using the job subsystem ■■ Perform configuration management to capture your current configuration, analyze it against referenced, saved, or live configurations, and drive to a certified configuration ■■ Generate easy out-of-box reports for users, managers, and executives using graphical report creation tools with secure publishing capabilities Chapter 1: Overview of the Grid Control Architecture 9 Grid Control gives you the ability to take control of a heterogeneous IT site that employs hardware platforms, network solutions, and databases from many vendors. Until recently, IT professionals—from the CIO down to the DBA—have only dreamed about centralizing control of all IT resources. One of Grid Control’s main objectives is to proactively monitor all systems so that DBAs can spend more time participating in high-level activities, such as planning the future technology direction of their IT departments. Back in the day before GUI applications, DBAs monitored systems at the command line with custom scripts. Grid Control standardizes such monitoring with out-of-box metrics to alert IT staff of problems with system resources and critical applications. Making Sense of Grid Control Versioning Oracle Enterprise Manager (OEM) 10g Grid Control versioning is confusing because it is not consistent. The following list shows the versioning scheme for all releases of the product since 10.1.0.2: Full Name of Release Release Family Release Number OEM 10g Grid Control Release 4 (10.2.0.4.0) EM 10gR4 10.2.0.4 OEM 10g Grid Control Release 3 (10.2.0.3.0) EM 10gR3 10.2.0.3 OEM 10g Grid Control Release 2 (10.2.0.2.0) EM 10gR2 10.2.0.2 OEM 10g Grid Control Release 2 (10.2.0.1.0) EM 10gR2 10.2.0.1 OEM 10g Grid Control Release 1 (10.1) EM 10gR1 10.1.0.6 OEM 10g Grid Control Release 1 (10.1) EM 10gR1 10.1.0.5 OEM 10g Grid Control Release 1 (10.1) EM 10gR1 10.1.0.4 OEM 10g Grid Control Release 1 (10.1) EM 10gR1 10.1.0.3 OEM 10g Grid Control Release 1 (10.1) EM 10gR1 10.1.0.2 As you can see, all versions of the first major release, Grid Control 10.1, are known as Release 1. However, the next major release, 10.2, actually refers to both 10.2.0.1 and 10.2.0.2 as EM 10gR2. Then, as of EM 10gR3, Release x describes the latest release family or patch set of 10.2. In other words, Release 3 (EM 10gR3) is 10.2.0.3, and Release 4 (EM 10gR4) is 10.2.0.4. This book covers these latest two releases which, combined, have been ported to every major platform. EM 10gR4 has already been ported to most major platforms, and EM 10gR3 is offered for the remaining ones. 10 Oracle Enterprise Manager 10g Grid Control Implementation Guide Grid Control Components Oracle Enterprise Manager 10g Grid Control is Oracle’s solution for managing your complete IT environment—Oracle and non-Oracle products alike. Grid Control gathers information about your enterprise computing environment, consolidating it into a central Repository. Grid Control displays this information to database administrators from its Web Console and sends them alerts on threshold conditions of interest. Administrators can then use this information to ask GC to perform tasks for the computing systems it monitors. Grid Control is built upon the Oracle technology stack: an Oracle database acts as the back end, while an Oracle Application Server is the glue of the middle tier. Let’s start by examining the main components of Grid Control. The basic GC topology consists of four core components, as depicted in Figure 1-2: the Grid Control Console, the Oracle Management Agent, the Oracle Management Service, and the Oracle Management Repository. Each component can be separated by a firewall. Note Throughout the book, I use the term “Grid Control infrastructure” or “Grid Control framework,” to refer to the three principal Grid Control architectural components, namely, the Oracle Management Repository, Oracle Management Service, and the Oracle Management Agent, which is installed on each target host. Although the Grid Control Console is indispensable to this framework, it is a “client.” Oracle Management Repository Thin JDBC Firewall Oracle Management Service HTTP(S) Firewall HTTP(S) Oracle Management Agent Grid Control Console FIGURE 1-2. Target Host Topology of Grid Control core components Chapter 1: Overview of the Grid Control Architecture 11 Following is a brief description of each core component. See Chapter 2 for a complete list of supported browser versions for the Console, and certified platforms for the GC framework, and check the OracleMetalink Certify tab for the most up-to-date information. ■■ Grid Control Console The Grid Control Console (Web Console or Console) is a browser-based Console in which administrators can centrally manage their entire computing environment. If you had to choose one image to define the big picture for Grid Control, it would have to be the Console home page. For now it’s enough to know that you’re looking through the front windshield of the Grid Control vehicle to get an overall view of your IT infrastructure. From the Console home page, you can turn down (i.e., drill down) any street of a specific managed target (system) within your IT infrastructure that Grid Control administers. The GC Console is certified to run on all the popular browsers, including Internet Explorer, Netscape, Mozilla, Firefox, and Safari. You don’t install the Console; the Management Service renders it. You just open a browser and connect to the Console via the GC login URL. Administrators can open as many Console connections as the GC site can accommodate from a performance standpoint, which is usually a very large number. 12 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ Oracle Management Agent The Oracle Management Agent (OMA or Agent), installed on each managed host, monitors that host and all targets on it, communicates information about these targets to the Oracle Management Service (OMS), and maintains the host and its targets as directed by administrators via their Console input. Targets are Oracle and non-Oracle products installed on a host. Grid Control monitors roughly 100 different target types out-of-box. For GC sizing purposes, each instance of a particular target type counts as a monitored target. Example target types are Database Instance, Listener, Oracle Application Server, and Host. You can also choose from among dozens of licesnable management options or “addons” to extend the list of non-Oracle products that GC can monitor. Grid Control monitors itself, so an Agent also runs on each node hosting the Oracle Management Service and the Oracle Management Repository. Each managed host runs one and only one Agent. You can have as many Agents as you can scale Grid Control to support. Oracle certifie the Agent on a large number of 32-bit and 64-bit host platforms, including AIX, Linux, HP-UX, Solaris, and Windows. ■■ Oracle Management Service The Oracle Management Service (OMS) is a Java 2 Platform Enterprise Edition (J2EE) middle-tier application that renders the user interface for the Console. (J2EE is an environment for developing and deploying enterprise applications). The Agent uploads target data to the OMS which then processes this data before uploading it to a data store, the Oracle Management Repository. The Grid Control middle-tier contains three elements: Oracle Application Server Containers for J2EE (OC4J), Oracle HTTP Server (OHS), and Web Cache. The OHS deploys the Management Service J2EE Web application. You must install the OMS on at least one host, but you can also install it on more hosts as needed to support your environment for scalability or high availability. Each Management Service should reside on its own host. The OMS and Oracle Management Repository can reside on the same host, but for performance reasons, Oracle does not recommend this configuration for a production Grid Control environment, unless it is small (see Chapter 2 for details). All physical OMS hosts logically provide the generic Oracle Management Service. At the time of this writing, the OMS was certified to run on AIX, HP-UX, Linux, Solaris, and Windows. ■■ Oracle Management Repository The Oracle Management Repository (OMR or Repository) is the data store for Grid Control created in an Oracle database, and is located in the SYSMAN schema. This schema contains information about all Grid Control targets, administrators, and managed applications. The Repository organizes this data so that the OMS can retrieve and display it in the Console for administrators. A Grid Control site uses just one central Management Repository database. It can be a single-instance or RAC Oracle 10g or Oracle 9.2 database, although Oracle 10g is highly recommended. In a nutshell, both the Console and the Agent communicate with the OMS. The following summarizes how these Grid Control components interact with each other (you can think of them as ad hoc and batch connections, respectively): Chapter 1: Overview of the Grid Control Architecture 13 ■■ The Grid Control administrator requests content in the Console over HTTP(S) in a browser session, which the OMS renders. The OMS then retrieves the data for the request from the Management Repository and displays it in the Console. ■■ Agents upload information to the OMS over HTTP(S), and the OMS uploads this data via JDBC to the OMR. The OMR sends data back to the OMS over JDBC, which is relayed to the Agent via a built-in HTTP listener. Now that you are familiar with the main components of the Grid Control architecture, let’s stop the engine, hoist Grid Control up on a lift, and take the mechanic’s flashlight tour. We would do well to examine these components more closely and identify their subcomponents. Such scrutiny will help illuminate how the components interact, and will allow you to trace the data flow between them for each type of Grid Control communication. Grid Control Console The Web Console provides the user interface to the Grid Control product. From any location with Web browser access, you can log in to the Grid Control Console and centrally manage your entire enterprise Grid environment. Because the Console interface is rendered in HTML, it uses HTTP (or HTTPS if secured), making it lightweight, easy to access, and firewall friendly. You can offer Console login through a portal. Wireless devices can also access a critical subset of Enterprise Manager functionality through its wireless component, EM2Go. EM2Go requires no additional setup, and is available using a wireless personal digital assistant (PDA), a combination of Bluetooth PDA and mobile phone, a PDA using a Wireless Local Area Network (WLAN), or Pocket PC Emulator. The usual way to log in to the Grid Control Console is at http[s]: : /em. The URL protocol (HTTP or HTTPS) depends on whether you configure Secure Sockets Layer (SSL) encryption for browser access to Console communications. (SSL is an Internet protocol that secures data transmission using public-key cryptography). You can log in via Web Cache or directly to the OHS. The URL port differs depending upon both the method of login (Web Cache or OHS) and the OMS platform (UNIX or Windows). Web Cache page performance is a little better, but it is useful to know how to bypass it, such as when trying to isolate Web Cache as the source of a Console login problem. In addition to the Web Console, there are several optional client components that work with Grid Control, and whose installation is covered in Chapter 8: ■■ Enterprise Manager 10g Java Console The EM Java Console is a thick-client supplied with the Oracle 10.2 Client software, containing select functionality still not converted to the Grid Control Web Console (thin-client). ■■ Adobe SVG Viewer This plug-in is required to display certain graphs in the Grid Control Web Console, which prompts you to install the plug-in if needed. ■■ EM Command Line Interface The EM CLI is for administrators desiring to access Web Console functions from an OS shell either interactively or directly from scripts, or to set up workflow for business processes. ■■ Oracle Configuration Manager Client OCM collects host configuration information to upload to Global Customer Services for analysis to better serve their customers. OCM is bundled with Grid Control 10.2.0.2 and can also be downloaded from OracleMetaLink as a standalone install kit. See Chapter 8 for details on these optional Grid Control clients, including how to install them. 14 Oracle Enterprise Manager 10g Grid Control Implementation Guide Let me speak briefly about the EM 10g Java Console, as many older DBAs are familiar with the OEM 9i Java Console. Almost all of the capability of the OEM 9i Java Console has been converted to HTML within the 10g Grid Control Web Console. To execute the few leftover 9i Java Console fat client features not yet available in the Grid Control 10g Console, such as management of Oracle Advanced Replication, you must download the Oracle10g Client software and install the Enterprise Manager 10g Java Console nondefault component. The EM Java Console only runs in standalone mode, optionally with its own standalone Repository; it does not connect to the Management Service or to the Management Repository. Oracle Management Agent An Management Agent must be installed on a host for Grid Control to monitor targets on that host. The Agent is the distributed portion of the Enterprise Manager framework, and is implemented in the C programming language for performance and resource reasons. It is a multithreaded process that uses Oracle core libraries, the Oracle Call Interface (OCI) and Oracle Secure Sockets Layer (SSL) to secure it by default. You run one and only one Management Agent on each host that has targets to be managed. A properly installed Agent automatically (i.e., out-of-box) starts monitoring itself, its host, any Grid Control components (OMR, OMS) installed along with the Agent on the host, and certain Oracle target types (databases, listeners, and Oracle Application Servers) already installed on the host. (The Agent and host are targets in their own right; Grid Control treats them as any other target.) This automatic target discovery begins as soon as the Agent is installed and starts up. Those targets not automatically discovered or installed subsequent to the Agent can be manually discovered in the Console. Targets can also be remote from the Agent to allow monitoring of targets without operating systems, such as firewalls and routers. In addition to the many built-in target types that Grid Control manages Oracle and its partners have built dozens of add-ons for non-Oracle products. Oracle-built add-ons are available for third-party products such as Microsoft SQL Server, Microsoft Active Directory, F5 BigIP, NetApp Filer, Dell PowerEdge Server, BEA WebLogic, and IBM WebSphere Application Server. Partnerbuilt add-ons include Citrix Presentation Server and Egenera pServer.7 For a complete list of all add-ons, see http://www.oracle.com/technology/ products/oem/extensions/index.html on the Oracle Technology Network. Note Grid Control can also monitor Oracle Collaboration Suite (OCS). Although Oracle uses OCS internally, this product does not have a sufficient market presence to warrant coverage in this book. The Management Agent uses default monitoring and data collection levels to provide out-of-box monitoring and management information about all discovered targets. The Agent immediately uploads metric alerts and periodically uploads management information to the Management Service. The Agent also performs tasks on behalf of the Management Service, from running a job (a unit of work defined to automate administrative tasks such as backups or patching) to setting blackouts (suspending data collection on one or more targets to perform scheduled maintenance). The Management Agent stores its configuration and management information about targets in flat OS files located under the Agent home directory. These files hold both unchanged, administrative 7 pServer is Egenera’s term for a virtual server. Chapter 1: Overview of the Grid Control Architecture 15 data and dynamic alert and metric data that the Agent generates from monitoring its targets. See Appendix A (online at www.oraclepressbooks.com) for a complete treatment of Agent files and directories and for a description of all Agent subsystems and how they interact. This appendix is meant primarily as a referral source when needing to delve into the internals of the Agent subsystems for troubleshooting purposes. However, it will also give you a general feel for how the Agent organizes itself at the file level to monitor targets. It will probably not make much sense to you until after you install all Grid Control components, including standalone Agents on target hosts, which isn‘t covered until Chapters 5, 6, and 7. Oracle Management Service The Oracle Management Service is the Grid Control middle-tier that renders the user interface for the Console. The OMS is a critical component of any Grid Control implementation. The Management Service not only receives upload information from Management Agents, but also sends data to and retrieves data from the Management Repository. The OMS also renders this data in the form of HTML pages, which are requested by and displayed in the client Console. In addition, the Management Service performs background processing tasks, such as delivering notifications and dispatching Grid Control jobs. The OMS is deployed on the Oracle Application Server. The EM10g installation begins by installing an instance of the installation type, “Oracle Application Server J2EE and Web Cache.” The OMS is deployed on its own OC4J container in this Application Server instance. Understanding the architecture of the OMS requires a grasp of each of its elements. The Management Service for EM10g Release 2 and higher comes bundled with Oracle Application Server Release 10.1.2,8 which consists of the following subcomponents: ■■ Oracle Application Server Containers for J2EE (OC4J) ■■ Oracle HTTP Server (OHS) ■■ OracleAS Web Cache Note The OMS is technically part of the OC4J, but throughout this book the entire GC middle-tier is collectively referred to as the OMS. You can deploy multiple Management Services, each on a separate host. However, the OMS on each host includes all subcomponents, and points to the same Repository database for a given Grid Control site. Each OMS communicates via Java Database Connectivity (JDBC) to the Management Repository. (JDBC is a standard Java interface for connecting from Java to relational databases.) For GC 10.2.0.3 or lower, Oracle does not support a Management Service installation on nodes running any kind of clusterware, including Oracle Real Application Clusters (RAC). This means that, unless running GC 10.2.0.4 or higher, it is not supported to install the OMS on a host containing a RAC database that houses the Management Repository. The following sections describe each GC middle-tier subcomponent. Oracle Application Server Containers for J2EE (OC4J) The OMS sits within an Oracle Application Server 10g Release 2 (10.1.2) environment, thereby benefiting from Oracle Process Manager and Notification Server (OPMN) control. (OPMN allows you to manage all OMS processes.) The other two OMS subcomponents, Oracle Application 8 EM10gR1 is bundled with Application Server 9.0.4. 16 Oracle Enterprise Manager 10g Grid Control Implementation Guide Server and Web Cache, also integrate with OPMN for process management, death detection, and failover for OC4J and OHS processes. As such, you should control all Management Service subcomponents using either the Oracle Application Server command-line tool, opmnctl, or EM Application Server Control, as discussed in Chapter 9. Oracle HTTP Server Oracle HTTP Server (OHS) is the web server for Oracle Application Server, and is based on the proven code base of Apache 1.3 Web server. Oracle HTTP Server, as part of Oracle Application Server, offers the following functionality to Grid Control9: ■■ Security ■■ Permits Grid Control to take advantage of the SSL encryption capabilities of Oracle HTTP Server to enable HTTPS security between browsers and the Grid Control Console. ■■ Allows GC administrators to define directives to limit access control (for example, by domain name and IP address). ■■ Provides the opportunity to integrate Grid Control with Oracle Application Server Single Sign-On (SSO) to authenticate GC administrators. ■■ High availability and scalability Provides these features through the use of multiple Oracle Management Servers, each with its own Oracle HTTP Server. ■■ Virtual directories Make URLs available for certain Grid Control functionality. For example, the Oracle HTTP Server includes the Management Service virtual directory http:// :4889/agent_deploy/ from which hosts to be managed can pull the agentDownload script and response file needed to install the Agent via this method. ■■ PL/SQL stored procedures Provide access to PL/SQL code stored in the Management Repository database through the Oracle module mod_plsql. ■■ PL/SQL Server Pages Allow Grid Control HTML pages to use PL/SQL code as the scripting language. HTML is translated into a stored procedure that uses the mod_plsql module to send the output to the Grid Control Console. ■■ Server Side Include Affords an easy way to add static and dynamic content across the Grid Control site (for example, the uniform static header and footer information that appears on most Console pages). ■■ Perl Supplies some dynamic content to Grid Control. The Oracle Application Server 10.1.2 bundled with Grid Control uses Perl 5.6.1. ■■ Dynamic Monitoring Service (DMS) Collects runtime performance metric statistics for both Oracle HTTP Server and OC4J processes. With this data, you can locate bottlenecks and tune application servers accordingly, though you should not tune the HTTP Server bundled with GC. However, DMS does supply Grid Control’s many HTTP Server metrics, including statistics on active connections, memory, process and CPU usage, request failures, and error rates. 9 Source: Oracle HTTP Server Administrator’s Guide 10g Release 3 (10.1.2), Chapter 1. I’ve adapted this material to explain how Grid Control takes advantage of generic Oracle HTTP Server functionality. Chapter 1: Overview of the Grid Control Architecture 17 OracleAS Web Cache OracleAS Web Cache (Web Cache) is a reverse proxy server accelerator, which is a server that appears as a content server to clients outside the firewall, but actually relays requests to back-end servers behind the firewall, then delivers retrieved content back to the client. This provides the following benefits for Grid Control10: ■■ Performance Reduces response times to Console requests by storing frequently accessed URLs in memory to avoid processing duplicate URL requests on the Oracle HTTP Server and Management Repository database. Not only does Web Cache cut down on the number of URL requests, its external cache also increases static throughput (logos, menus, tabs, and header/footer information) of the Grid Control Web site, processing these requests 10 to 100 times faster than application-specific object caches. Note that over 90 percent the data on GC Console pages is dynamically generated, and does not benefit from increased Web Cache throughput. ■■ Scalability Allows for many more concurrent Console connections, thereby reducing Oracle HTTP Server errors. The increased throughput itself also provides scalability. ■■ Cost savings Provides cost savings for Grid Control implementations because of the increased performance and scalability. Administrators need fewer Oracle Management Servers (i.e., fewer HTTP Servers) for the same Console load. OracleAS Web Cache behavior varies depending on whether a request from a GC administrator results in a cache hit or a cache miss. Here’s how it works. An administrator logged into the GC Console through Web Cache sends an HTTP or HTTPS request for content. Web Cache acts as a virtual server on behalf of the Oracle HTTP Server for its Management Service. If any part of the content is in its cache (such as from a previous navigation to this page), Web Cache sends this part of the content directly to the Console browser and stores a copy of the page in cache. This is a cache hit. If Web Cache does not have the requested content or if the content is stale or invalid, it hands off the request to the Oracle HTTP Server, which requests this data over HTTP or HTTPS from the Management Service. This is a cache miss. Oracle Management Repository The Oracle Management Repository is the comprehensive source for all the management information for Grid Control. It consists of schema definitions, stored procedures, and RDBMS jobs within an Oracle database, all owned by the SYSMAN database user. A particular Grid Control implementation employs only one Management Repository. You can install multiple Management Services, but each must use this central Repository as their data store. All Grid Control Console administrators share the same Repository information based on the priviledges granted to them. Information in the Repository includes: ■■ Configuration details about the managed targets ■■ Managed target availability information 10 Source: Oracle Application Server Web Cache Administrator’s Guide 10g Release 2 (10.1.2), Chapter 1. I’ve adapted this material to explain how Grid Control takes advantage of generic Web Cache functionality. 18 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ Historical metric data and alert information ■■ Target response time data ■■ Inventory information on patches and products installed This information allows administrators to manage their complete application stack databases, application. The Management Repository can reside on either the same host as a Management Service or on a dedicated host. You make this decision on the first screen of the Grid Control installer, where you select one of the first two Installation Types: Enterprise Manager 10g Grid Control Using a New Database or Enterprise Manager 10g Grid Control Using an Existing Database. When you choose the first installation option, the installer creates a new 10.1.0.4 database, and lays down a new Repository, OMS, and Agent, all on the local host. As for the second installation option, the OMS can be placed on either the OMR host or a separate dedicated host—you run the installer on the host where you want the OMS to reside. Oracle supports running the Repository in either a single-instance or Real Application Clusters (RAC) database, Release 9.2 or 10g. You must locate the Repository in an Enterprise Edition database and must select the Partitioning Option because GC uses partitioned tables to store management data. Now let’s examine the Repository schema, including the schema owner, tablespaces, and objects within that schema. SYSMAN is both the schema owner and the default Super Administrator account. You cannot remove or rename the SYSMAN account. It can be used to: ■■ Create GC privileges and roles ■■ Perform the initial GC setup, such as creating administrator accounts and notification rules ■■ Discover new targets ■■ Create generic jobs to run on all databases or hosts The Grid Control installer creates two default tablespaces to hold its objects: ■■ MGMT_TABLESPACE This tablespace holds all management information except configuration management data. ■■ MGMT_ECM_DEPOT_TS This tablespace stores configuration management data, which includes support for Binary Large Objects (BLOBs). Aside from the benefit of logically isolating configuration data in a separate tablespace, it is a best practice for performance and tuning reasons to locate BLOBs in a separate tablespace. As for the schema objects, an open Repository schema is the key to one of Grid Control’s most important architectural features: extensibility. An open schema means that you can customize the use of Repository data if the standard configuration does not meet your requirements. The Repository tablespaces contain base tables and indexes that begin primarily with MGMT_ and other data types, including over 3000 database views. (Views are virtual object tables that offer specialized or restricted access to the data and objects in a database. Views provide the flexibility to look at the same relational or object data in more than one way.11) The views, whose names start 11 Oracle Database Concepts 10g Release 2 (10.2), Chapter 27. Chapter 1: Overview of the Grid Control Architecture 19 mostly with “MGMT$”, are particularly handy for mining Repository information that you want to process further. The views are comprehensive so that you can avoid having to directly access the base tables. The inherent advantage to these views is they insulate custom applications from underlying changes to the base SYSMAN schema due to new releases or patching. Caution Views supplied with Enterprise Manager 10g Release 10.2 are “provisional.” This means that while Oracle fully supports these views for Release 2, they may change in subsequent releases and are not guaranteed to be backward compatible. Following are the categories of GC views that provide access to metric, target, and monitoring data stored in the Repository: ■■ Monitoring ■■ Inventory ■■ Policy definition ■■ Policy association ■■ Policy violation ■■ Management template ■■ Job ■■ Application Service Level Management ■■ Configuration ■■ Oracle home patching ■■ Linux patching ■■ Security ■■ Configuration management ■■ Database cluster ■■ Storage reporting Views provide the basis for passing alerts on to other System Management products, for customizing new add-ons, reporting, performing historical analysis, and data computation. Data Flow Between Grid Control Components Now that you know all GC components and subcomponents, it’s time to connect the dots between components to understand how they interact, even down to the subcomponent level as is required to do justice to the Grid Control architectural model. Figure 1-3, 6 diagrams the internal workings of this engine to show how data flows between components for both Console communications and Agent uploads of metric data and alerts. 20 Oracle Enterprise Manager 10g Grid Control Implementation Guide Managed Host TCP/IP Managed Target Grid Control Console 7c Oracle Management Agent 2a 7b 7a https://OMAhost:3872/emd/main Oracle 1 http://OMShost:7777/em Management Service OracleAS Web Cache 2b HTTP(S) OMS monitors OMA Metric data https://OMShost:1159/em/upload 3b HTTP(S) HTTP 3c Oracle HTTP Server 4 SQL*Net traffic on Database Listener Port Ping Manager (HTTP heartbeat) HTTP(S) Oracle Application Server Containers for J2EE (OC4J) 6 5 Oracle Management Repository FIGURE 1-3. JDBC Target metadata changes and state information 3a Data flow for interactions between Grid Control components Let’s follow the numbers and arrowed lines in this figure to visualize the flow of Console and management data requested from one component or subcomponent to another. Default ports are listed in the figure. However, an administrator can customize all ports within the GC infrastructure as a specific site’s environment requires. 1. An administrator logs in to the Console at http[s]:// : /em. Port 7777 shown in the figure is the Web Cache nonsecure login port for UNIX. 2. The administrator requests content in the Console, prompting two possible responses, depending on whether the requested content is found in Web Cache (this is irrespective of whether you log in via Web Cache): a. If some of the requested content is in its cache (such as from a previous navigation to this page), Web Cache sends this content directly to the Console browser. Chapter 1: b. Overview of the Grid Control Architecture 21 If Web Cache does not have any of the requested content or if the content is stale or invalid, it hands off the request to the Oracle HTTP Server (2b), which requests this data over HTTP or HTTPS from the Management Service (4). The OMS then requests this data via JDBC from the Repository (5), and, upon receipt, renders the content in the Console from (6) to (4) to Console. 3. An Agent communicates with both its OMS and the Repository. The Agent uploads its information directly to the Repository or through the Oracle HTTP Server of the OMS to the Repository, depending on the criticality of the Agent communication as defined by the Agent channels (see “How Agent Files Are Uploaded” in the online Appendix A). The Agent also sends a periodic heartbeat called an Agent ping to its OMS indicating that it is available. Here are the specifics: a. The Agent connects directly to the Management Repository to report target metadata changes (contained in A channel files) and state information (stored in B channel files). b. The Agent uploads metric data (C and D channel files) to the Oracle HTTP Server URL (unlike administrator requests, which first go through Web Cache if they’re logged in that way). The default HTTP Server URL is https:// :1159/em/ upload for secure HTTPS uploads. (The Agent uses http:// :4889/em/ upload for unsecured HTTP uploads and wallet requests to secure the Agent; even when the Agent is already secured, the unsecured port must still be accessible to allow an Agent to re-secure if necessary.) c. The Agent heartbeat sent to the OMS indicates that the Agent is running. 4. The Oracle HTTP Server uploads Agent data over HTTP or HTTPS to the OMS. 5. The OMS uploads Agent data via JDBC to the Repository. 6. If the Repository needs to send data to the Agent, it uses JDBC to send information to the OMS to relay to the Agent via its built-in HTTP listener. 7. The OMS communicates to the Agent in several ways: a. The Management Service forwards data directly to the Agent over HTTP. As mentioned earlier, no Oracle HTTP Server is required on the Agent host because the Agent software includes a built-in HTTP listener listens on http:// :3872/emd/main. The OMS also submits jobs and other management tasks through this URL. b. The OMS uses the Agent URL to monitor the status of the Agent. (If the OMS-to-Agent communication is not successful, the OMS pings the host on which the Agent resides with ICMP Echo requests.) c. The OMS sends SQL*Net traffic over TCP ports to target database listeners on Agent hosts, such as when patching database targets through Grid Control. Now that you’ve looked more closely at the internal workings of the Grid Control engine, you’ll have a better appreciation for it when you fire up the “stock” engine in Chapter 3. With enough detailed instructions, even a novice mechanic can install the base GC product. You’ll 22 Oracle Enterprise Manager 10g Grid Control Implementation Guide Grid Control Weaves into a Grid Infrastructure and Monitors Itself Now that you have studied the GC components, you can plug them into a grid infrastructure and use Grid Control to monitor that infrastructure, including its own components. Yes, Grid Control can actually monitor itself. Think about it. The Repository is housed in a database, which can be one of the databases supported by the RAC Cluster in Tier 3 (refer back to Figure 1-1). The application tier (Tier 2) can host the OMS. The Console is a browser on a client workstation, like any other client in Tier 1. Agents on all grid hosts monitor all grid components as targets, including targets without operating systems (switches, routers, load balancers, etc.) monitored by remote Agents, and the Grid Control components and subcomponents themselves.12 While it is the best practice to weave Grid Control into a grid computing environment, Grid Control monitors itself irrespective of your adherence to a grid computing architecture. have plenty of time before the tougher high availability custom job in Chapter 15 rolls into your garage.12 Grid Control, Database Control, and AS Control For EM 10g, the term “Enterprise Manager” encompasses Grid Control, Database Control, and Application Server (AS) Control. The word on the street is that the release of EM 11g is taking an ambitious “EM Everywhere” approach to eliminate the distinctions between the “Controls”. For EM 10g, however, we must maintain this distinction. Think of Database Control as Grid Control for just one database. Therefore, the databaserelated material in this book, with few exceptions, applies equally to Database Control as it does to Grid Control. Application Server Control administrative functionality, on the other hand, is not encompassed in Grid Control, but is part of the Oracle Application Server. Grid Control does monitor AS Control, and simply provides the administrative link to AS Control for the AS targets it monitors. While this book touches on Application Server Control as it relates to Grid Control, for more in-depth coverage of AS Control, consult the Oracle documentation for Oracle Application Server or refer to the excellent book Oracle Application Server 10g Administration Handbook, McGraw-Hill/Osborne, 2004. Note This book narrows the scope of Enterprise Manager coverage to just Grid Control. It does not address Database Control or Application Server Control per se, although most of the target database-related material applies just as much to Database Control as it does to Grid Control. What differentiates Grid Control from Database Control and from Application Server Control? 12 How can Grid Control send you notification that the Repository or OMS are down when they must be up to send such notification? For the answer, see the section “Enable Notification if Grid Control Goes Down” in Chapter 11. Chapter 1: Overview of the Grid Control Architecture 23 Grid Control vs. Database Control What is the difference between Oracle Enterprise Manager 10g Database Control and Oracle Enterprise Manager 10g Grid Control? Database Control is a web-based application (like Grid Control) installed with Oracle Database 10g. From the Database Control Console, you can only administer the accompanying single instance or RAC Oracle Database and ASM instances the database uses for storage.13 The Database Control architecture is very similar to that of Grid Control but on a smaller scale. The equivalent of the Grid Control Central Agent and OMS are rolled into the same OC4J application, and the Repository is located in the target database itself. This means that Database Control does not work if the target database is down, except that it can start the database. The Database Control home page has the same look and feel as the Grid Control home page. To take an example, here is a home page for a cluster database named ptc. It is clear from this home page that this Database Control Console only manages this particular cluster. By contrast, from the Grid Control Console, you can centrally manage multiple databases 13 Database Control also gathers some host statistics for the server on which the database resides. Therefore, you can also monitor and manage the host with Database Control. 24 Oracle Enterprise Manager 10g Grid Control Implementation Guide and many other types of targets as well. It is best practice to run either Grid Control or Database Control, but not both, although you can run both concurrently. Grid Control vs. AS Control How is Grid Control different from Application Server (AS) Control? In short, as just one of many target types it monitors, Grid Control monitors the Oracle Application Server, whereas AS Control administers the Oracle Application Server. Grid Control Monitors Oracle Application Server While Grid Control administers all Oracle products in your enterprise, it only monitors the Oracle Application Server—both the AS bundled with Grid Control and any separate AS targets that Grid Control monitors. Monitoring AS instances in the Grid Control Console encompasses real-time monitoring, alerting, and historical data collection, just as with any other Grid Control target. For example, the Application Servers tab within the GC Console reveals the status, alerts, policy violations (noncompliance with a desired state for security, configuration, or storage), and resource usage of all Application Servers and their subcomponents, as shown here. Chapter 1: Overview of the Grid Control Architecture 25 AS Control Administers Oracle Application Server By contrast, use the standalone AS Control Console to administer AS subcomponents. To get a better understanding of what it means to administer the Application Server, take a look at the AS Control Console home page, accessible directly at http:// :1156 on UNIX or http:// :18100 on Windows, or through the Grid Control Console on the Application Server home page using the Administer link. From the AS Console home page, you can control OMS and non-OMS Application Server subcomponents—the Oracle HTTP Server, OC4J instances, and Web Cache. This control is from A to Z; you can stop, start, restart, enable, or disable subcomponents. You can also gather information about the management software itself by clicking the Management link at the bottom of the System Components section on the Application Server home page. The AS Console has four other tabs: J2EE Applications (which allow you to view response times for OC4J instances), Ports (which you can change through the UI), Infrastructure (including Identity Management and OracleAS Farm Repository Management), and Backup/Recovery (of AS data and configuration files). Without going into more detail, as Application Server Control is beyond the scope of this book, I hope I’ve given you just enough information to make the distinction clear between Grid Control and Application Server Control. Oracle TIGHT / Oracle Enterprise Manager 10g Grid Control Implementation Guide / Michael New / 149275-5 26 Oracle Enterprise Manager 10g Grid Control Implementation Guide Agents for Grid Control, Database Control, and AS Control Now that we are clear that Grid Control, Database Control, and Application Server Control are all distinct, you’re probably not surprised to hear that separate Agents exist for each. The Grid Control central Agent on each managed host (including OMS and OMR hosts) monitors all targets on that host and uploads target information to the SYSMAN schema in the Management Repository. Note In this book, Management Agent or Agent refers to the central Grid Control Agent. The Database Control Agent, on the database host, similarly stores data in a SYSMAN schema, but this schema is local to its associated Oracle database. In contrast to both Grid Control and Database Control Agents, the Application Server Control Agent, bundled with the Oracle Application Server, only communicates real-time data internally to the Application Server Control, as there is no Repository for historical data. All three Management Agents are distinct. They run from their respective home directories—the Grid Control Agent from its own Agent home, the Database Control Agent from the database home, and the Application Server Control Agent from the Application Server home including that for the OMS. Grid Control, Database Control, and AS Control: All Together Now Let’s build on the topology of Grid Control core components shown earlier in Figure 1-2 by adding Database Control and Application Server Control to the mix, as represented in Figure 1-4 (I’ve omitted protocols for clarity). I have also included four typical managed targets: a 10g Database Server, a 10g Application Server, an 8i/9i Database Server, and a third-party application. For good measure, the figure illustrates two OMS hosts, with two of the central Agents uploading management data to one of the OMS hosts, and the remaining two Agents uploading to the other OMS host.14 In addition, a Grid Control Console communicates with an OMS, which also receives Agent uploads of target metric data. Although each OMS only receives data from the particular Agents that report to it, each OMS uploads this data to a central Management Repository, which in turn makes information from all Agents available to each OMS. This allows any GC Console to administer or monitor all Grid Control targets. On the other hand, a particular Database Control Console is in contact with just one database, and likewise, a particular AS Control Console connects with just one Oracle Application Server. The Database and AS Consoles are privy only to local database and Application Server data, respectively. All three Consoles can operate concurrently, but (as already stated) it is best to shut down Database Control when using Grid Control, as Grid Control can administer each of the databases to which a particular Database Console has access. 14 In a high availability configuration, as discussed in Chapter 15, rather than assigning Management Agents to particular Management Services, you would use a hardware server load balancer to virtualize the Management Services and shared storage to allow for failover between them. Chapter 1: Overview of the Grid Control Architecture 27 Oracle Management Repository Firewall Oracle Management Service Oracle Management Service Administration Administration Administration Monitoring Central Agent Central Agent Central Agent Central Agent Grid Control Console Grid Control Console Oracle 10g Database Server Monitoring 10g Database Control Console FIGURE 1-4. Firewall Oracle 8i/9i Database Server 3rd-Party Application Database Control Agent Oracle 10g Application Administration Server AS Control Agent Managed Targets 10g Application Server Control Console Grid Control, Database Control, and AS Control Each Application Server Control Console should remain available, however, to administer a particular Application Server, because Grid Control can only monitor an Application Server. I hope all this makes the distinction clear between Grid, Database, and AS Control. Summary Here is a synopsis of what was covered in this introductory chapter: ■■ We would be remiss if we made no mention of grid computing technology, which Grid Control was designed to administer. I began this chapter by presenting a brief overview of grid computing. ■■ Next, I provided a brief history of Grid Control and highlighted several product features. ■■ From there, I introduced you to the four main Grid Control components: the Console, Agent, OMS, and Repository. 28 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ I then broke down the main GC middle-tier components into subcomponents, and examined the interaction between all GC components. You saw how components work together to satisfy Console requests and upload metric data from the Agent on a target host through the OMS to the Repository, and how the OMS in turn delivers data back to the Agent. ■■ I ended this chapter with an ancillary but critical explanation of how Grid Control differs from Application Server Control and Database Control. This delineation helps further refine the scope of the book to just Grid Control and Application Server Control as it relates to Grid Control. Now that you understand the basic architecture and concepts of Grid Control, we can proceed to Chapter 2, where we’ll meet in the garage for preinstallation. Then, in Chapter 3, we’ll install Grid Control and get this thing on the road. Chapter 2 Grid Control Preinstallation 29 30 Oracle Enterprise Manager 10g Grid Control Implementation Guide I n Chapter 1, you got a feel of the workings of the Grid Control “engine” by becoming familiarized with its components (and subcomponent) and examining how they work together to process Agent management data and Console requests. In this chapter, we prep the engine, and in Chapter 3, we begin building it. The preinstallation steps discussed here are necessary to install a basic GC environment, which consists of an Oracle Management Repository (OMR) database (single-instance or RAC) servicing one or more Oracle Management Service (OMS) hosts that in turn talk to Oracle Management Agents (OMAs) on each target host. You will prepare to install the OMR database and one OMS either on the same host or on separate hosts, in close proximity to one another (i.e., in the same data center and preferably on the same subnet). I also provide sizing guidance and instructions for deploying additional Active OMSs on separate hosts. Note This chapter is for setting up Active/Active OMS and Repository (i.e., RAC) installations. See Chapter 15 now if interested in implementing an OMS or OMR Active/Passive configuration using Cold Failover Cluster (CFC) as each requires a special Grid Control installation procedure. Let’s take a look at how the preinstallation tasks are organized in this chapter. They fall into the following four major categories: ■■ Architectural design The first preinstallation step is to make the minimum number of design decisions required to install a basic GC environment, which can serve as a common denominator for any high availability and disaster recovery topology to be implemented when we get to Chapter 15 (except CFC configurations, which you must implement when installing GC). ■■ Network configuration After fleshing out the basic architecture, confirm the network configuration, including host naming rules and constraints and connectivity checks between GC component nodes. ■■ Hardware requirements Provide the hardware resources (disk space, RAM, swap, and CPU speed) for OMR and OMS hosts to meet Grid Control installation and operating requirements. ■■ Software requirements Confirm that GC meets all certification standards, create the necessary OS groups, users, and directories, stop listeners on port 1521, synchronize OS timestamps/time zones, and satisfy all platform-specific software requirements. The preinstallation tasks in this chapter apply to the hosts where the OMR, OMS, and chaininstalled Agents will be installed, as well as to hosts where standalone Agents will be installed. Chapter 2: Grid Control Preinstallation 31 Stage the Grid Control Software OMR, OMS, OMA In the inimitable IT multitasking fashion, I advise that you begin downloading and staging the GC software while carrying on with the remaining preinstallation steps. You’ll need to stage the distribution soon enough anyway to run the prerequisite checker, as described shortly. To stage the GC software, you must obtain it, then set up access to run the installer on the chosen host. The method of access varies depending upon whether the target system is local or remote. There are two ways to obtain the Oracle Enterprise Manager 10g Grid Control distribution software for a particular platform: download it from OTN or order the Grid Control Media Pack from the Oracle Store. (Even if you order the Media Pack, you still need to download the latest Patch Installer, which is not included.) ■■ Download from OTN Download the latest full GC software release for your platform from the Oracle Technology Network (OTN) Web site at http://www .oracle.com/technology/software/products/oem/index.html. Make sure to select the link under the Full Installers section for the latest full installation available for your platform.1 The Full Installers contain the OMR, OMS, OMA, and Management Packs, whereas the Patch Installers only contain the OMS, OMA, and Management Packs. No platforms currently offer Full Installers for the latest GC release. Therefore, you need to download the Full Installer archive for an earlier release, and the Patch Installer archive for the latest release (applied in Chapter 4). The Full Installer archives consist of one or more zip files, depending upon the platform, ranging from 1.3G on Windows to 4.3G on HP-UX Itanium. Download all zip files and unzip them into the same staging directory (it creates its own directory structure). ■■ Order the Media Pack Alternatively, order the latest Enterprise Manager 10g Grid Control Media Pack for a particular platform from the Oracle Store for a nominal fee. To order the Media Pack, go to http://oraclestore.oracle.com, choose your country, click the Media Packs link on the left navigation pane, select the platform, select the Enterprise Manager Media Pack, click Add to Cart, Checkout, and follow the checkout procedures. The Media Pack for each OS contains the base EM distribution and the Monitoring plug-ins (discussed in Chapter 11). When you receive the Media Pack, you can either run the DVD-ROMs directly or stage them on disk. Whether you download the distribution software or order the Media Pack, you also need to download the Agent software for managed host platforms other than your OMS platform. Download this Agent software under the Mass Agent Deployment section of the OTN site indicated above for Enterprise Manager. Once you obtain the GC software from either one of these sources, you can access the software via several methods, depending on whether the target system is local or remote. (Continued ) 1 1 I assume here that you are doing a fresh GC installation. If you need to upgrade a GC 10.2.0.X or 10.1.0.X installation, see the latest Patch Installer README or release notes for your platform. 32 Oracle Enterprise Manager 10g Grid Control Implementation Guide If the target system is local, access the GC software directly on that system. On Windows platforms, no further action is required to run the installer locally. On UNIX platforms, run the installer either from an Xwindows session or from a UNIX workstation. If the target system is remote, access the GC software from a remote DVD drive or use remote access software, as follows: ■■ Access the GC software from a remote DVD drive (UNIX only). If the system where EM is to be installed does not have a DVD drive, you can share a remote DVD drive. ■■ Access the GC software using remote access software. If you don’t have physical access to a remote system where installing GC, but this system has a local hard drive, you can install GC using remote access software such as Microsoft built Remote Desktop Connection, VNC from RealVNC Ltd., or Hummingbird Exceed. Start the remote access software on both your local computer and the remote system and install GC in one of the following ways: ■■ Copy the GC software to a hard drive and install from that hard drive. To do this, share the hard drive, then using the remote access software, map a drive letter on the remote system to the shared hard drive. ■■ Access GC software from a DVD drive in your local computer. Insert the DVD into a drive on your local computer and share the DVD drive. Use the remote access software to map a drive letter on the remote system to the shared DVD drive. You are now set up to run the Oracle Universal Installer (OUI), as described in Chapter 3, from the shared hard drive on the remote host using the remote access software. On most UNIX systems, a DVD disk mounts automatically when inserted into the disk drive. In case the DVD is not automatically mounted, the following lists the command to set the mount point for each UNIX platform (execute as root): AIX HP-UX Red Hat Linux SUSE Linux Solaris /usr/sbin/mount -rv cdrfs /usr/sbin/mount -F cdfs -o rr mount -t nfs :/ mnt/ mount -t nfs :/ media/ /usr/sbin/mount -r -F hsfs See the platform-specific GC documentation for more detailed instructions on mounting the product disks. Chapter 2: Grid Control Preinstallation 33 How do chain-installed Agents (as referred to in the Oracle EM documentation) differ from standalone Agents? ■■ Chain-installed Agents You do not explicitly install these Agents; their installation is bundled with that of an OMS . The Agent accompanies the OMS, “tags along for the ride,” so to speak, to monitor the OMS as well as the Repository (if installed on the same host) and any other host targets. Chain-installed Agents don’t have any distinct prerequisites beyond those for the OMS they accompany. ■■ Standalone Agents You explicitly install these Agents on each target host (except on OMS hosts where Agents are chain-installed) using one of six methods covered in Chapters 5, 6, and 7. Some of the preinstallation steps for the OMS also apply to standalone Agents, as indicated in this chapter. Standalone Agent and chain-installed Agent installations, once completed, are identical. Each plays the role of a central Agent that GC needs to manage a particular target host. In this chapter, I annotate all preinstallation tasks with OMR, OMS, and/or OMA (for any Agent installation method) to indicate on which host(s) to execute the task. Complete all OMR and OMS requirements now on the OMR node(s) and OMS hosts, respectively. (You can repeat the OMS tasks covered in Chapter 15 for any additional OMS hosts you may install then for high availability reasons.) You can either complete the standalone OMA requirements on all target hosts concurrently, or wait and circle back to them when you get to Chapter 5, which lists additional prerequisites for standalone Agents. Most administrators are anxious to install the OMR and OMS and don’t want to concurrently perform certain preinstallation steps for standalone Agents on all target hosts. If you feel this way, then, as the wizard says in The Wizard of Oz, “Pay no attention to that man behind the curtain,” i.e., “Pay no attention to those standalone Agent requirements in Chapter 2.” (My quote doesn’t have the same ring to it, but you get the idea.) You can certainly wait to take care of these Agent requirements all at once when you get to Chapter 5. Don’t worry; I will remind you. Note Again, the OMA annotations below indicate standalone (i.e., central) Agent preinstallation steps for any Agent method. You can elect to postpone these steps until Chapter 5, as they are not required for the Agent chain-installed with the OMS in this chapter. While many of the preinstallation steps apply generically to all operating systems, the commands to verify or fulfill some of these steps differ by platform. Where the syntax varies, I provide the variations for Windows and all flavors of UNIX. Key Architectural Design Decisions Grid Control preinstallation starts with design. This book splits that design into two stages: build a stock GC engine (described in this chapter), then “sup’ up” this engine to meet high availability (HA) and disaster recovery (DR) requirements (covered in Chapter 15). 34 Oracle Enterprise Manager 10g Grid Control Implementation Guide Here are a handful of few reasons to justify this two-tiered approach for those IT professionals who are skeptical by nature (i.e., for all you IT professionals): ■■ It’s a daunting task for a Grid Control newbie to tackle a full-blown implementation using all the HA and DR bells and whistles. On the other hand, it’s very attainable for an experienced DBA not completely familiar with Grid Control to make a few easy preliminary design decisions to implement the base product according to step-by-step instructions. Supplying such instructions is the major intent of this chapter. ■■ Oracle shops with little or no production monitoring in place understandably want to get GC up and running as quickly as possible to start monitoring their systems. It’s better to install the base GC product quickly to reap the benefit of immediate out-of-box monitoring than to continue without monitoring while you design and implement a high availability GC solution. ■■ You can benefit from GC data about itself to better compute its availability and performance as measured against SLA requirements. From this data, you may determine that the additional OMS host or RAC Repository node you planned to add isn’t necessary after all. Even a stock GC engine has built-in options. Choose those options now that you absolutely need, and leave the high-end stuff for later. You don’t need to know the inner workings of a sports car engine to buy one and drive it off the lot. You only need to answer three key questions now to install a basic GC system in Chapter 3: ■■ How many Grid Control environments should you build? ■■ How many regional sites are required? ■■ What installation choices should you make? Before addressing these questions, let me define what I mean by a GC site as opposed to a GC environment. A particular GC environment is comprised of one or more sites.2 A Grid Control environment is a system consisting of one or more coupled OMS and OMR hosts (single-instance or RAC) that manage Agents on target hosts. Targets on these hosts can be production, nonproduction, or both, depending on whether you decide to monitor production and nonproduction targets with separate GC environments. Regardless of this decision, the targets for a particular environment are logically related in that you want to manage all of them with just one GC system, if possible. Grid Control sites are mini-environments (each with an independent set of OMS, OMR, and OMA hosts) that are logically coupled with other sites to provide the geographical coverage necessary to service a particular environment. Network limitations or other considerations require you to install multiple sites. The sites are logically, not physically, related so they are really environments in their own right. 2 In this chapter, I distinguish between a GC “environment” and a GC “site.” However, in other chapters, I use the terms interchangeably. Chapter 2: Grid Control Preinstallation 35 Decide How Many Grid Control Environments to Build OMR, OMS Before you can prepare OMS and OMR hosts for installation, you need to select the host or hosts where these components will reside. The first question that springs to mind is whether this first (and perhaps only) GC environment will monitor production targets, nonproduction targets, or both. Most companies want to monitor nonproduction targets, too, so they must choose one of the following GC instance management strategies: ■■ Build at least two GC environments, one for production targets and one or more environments for nonproduction targets ■■ Build one GC environment for both production and nonproduction targets Note Your instance management strategy may call for multiple nonproduction environments, such as testing, pre-production, and development. However, most companies do not run more than one nonproduction environment. A GC implementation is itself a production target if it monitors production targets. As such, best practice is the first option: configure a production GC environment to monitor only production targets. This prevents nonproduction targets from adversely impacting a production environment. “Shorts” can occur anywhere in the “wiring” that connects components: from the Agent to the OMS, and from the OMS to the Repository: ■■ Agent to OMS On a host with both production and nonproduction targets, the Agentto-OMS upload process can stall on a file containing nonproduction data that must be synchronously loaded, i.e., data that blocks all other Agent uploads. These queued Agent uploads could contain critical production alerts whose notification would be delayed by a “stuck” nonproduction data upload. ■■ OMS to OMR The collection of nonproduction target data can cause a Repository database deadlock or other unexpected conditions, thereby causing an OMS-toRepository loader backlog of Agent data.3 A prerequisite for isolating production and nonproduction GC management is that target hosts exclusively contain either production or nonproduction targets. Only one Agent can run on a given host (perhaps a virtual host), and this Agent can only belong to one GC environment. If, as a rule, production and nonproduction targets are allowed to “cohabitate” on the same servers, you may as well let one GC environment manage both types of targets. 3 A salient case in point, since fixed in EM 10.1.0.4, was Bug 3785480 where a metric query that took a long time to execute due to a row exclusive blocking lock on a Repository table resulted in a backlog on the OMS tier of Agent files to be uploaded to the Repository. Some Oracle customers opted to completely shut down all production Agents until the bug was fixed. 36 Oracle Enterprise Manager 10g Grid Control Implementation Guide Caution Oracle does not support running more than one Agent on a single host, except in Active-Passive environments that run third-party Cold Failover Cluster (CFC) software.4 While it may be feasible to do so, such as by changing the default ports of the second Agent installation, it makes no sense as a way to separate production from nonproduction environments. If a host contains both production and nonproduction targets, you’ve already intertwined these two environments in that they both must contend for common hardware resources. Installing separate Agents would only compound the problem since multiple Agent installations would have to vie for the same hardware resources. This advice that, with regards to production and nonproduction targets, “neither the twain shall meet” also applies to production OMS and OMR hosts and nonproduction targets. In fact, try to avoid placing even production targets on hosts running production OMS or OMR components, as it’s not possible to predict possible resource contention. Any target that consumes undue resources on a GC infrastructure host can cripple or even bring down that host, thereby jeopardizing the monitoring of all targets. If you’ve decided on separate production and nonproduction GC environments, it’s standard practice to build the nonproduction environment first, just as it would be for any software application. If you plan to monitor all targets with one environment, then consider this chapter the beginning of a production build. Either way, you need to earmark one to three servers for your installation. Decide now whether these servers are for a production or a nonproduction GC environment. In the following two sections, you will determine the number of servers required and the GC components to be installed on each server. How Many Regional Sites Are Required? OMR, OMS A question that goes hand in hand with whether to build separate production and nonproduction environments is “How many sites are geographically required for a particular GC environment?” More aptly put, what is the fewest number of GC sites you can get away with for a given environment? For instance, if all production targets are on the east coast of the U.S. and all nonproduction targets are on the west coast, network bandwidth constraints due to geography may require building two regional sites, one for each environment. Here I define a GC “site” as a logical entity. One or more GC sites comprise a particular GC environment. Companies deploying Grid Control install one or more regional sites to provide complete geographical coverage for an environment, which is defined in terms of managing production targets, nonproduction targets, or both. Sites are logically related, but are physically no different from environments. GC sites/environments are physically unrelated to other sites/ environments; that is, all components—the constituent Agents and OMSs reporting to a Management Repository—for a particular site/environment are separate from any other site/ environment. The difference is purely on a logical level. 4 In CFC environments, you must install one Agent to each physical node of the cluster and another Agent using the logical host name of the cluster. Chapter 2: Grid Control Preinstallation 37 Barring the instance management strategy dictating the number of environments, Oracle’s direction, and that of the IT industry in general, definitely favors a consolidation strategy of using just one site to manage each environment. The technology direction for Grid Control consolidation is no exception. Grid Control has been implemented globally, but only at firms with fast networks. To calculate whether you can get away with one global site for a particular environment, you need to know the network requirements between GC components. It turns out that network performance between the OMS and OMR dictates how many GC sites each environment requires. Fortunately, Grid Control network error handling is robust, allowing it to tolerate network glitches and outages between components. However, Agent/OMS and Console/OMS network issues impact GC performance much less than network performance between OMS and OMR tiers. A particular Agent to OMS link may even be severed for a time without being noticed at the system level. Of course, certain administrators may be left hung out to dry if the network connection slows between the Console and a particular OMS. However, a network problem between OMS and OMR tiers has more global implications, because it could significantly reduce overall system performance. Console connection performance would deteriorate, along with monitoring (alerts and notifications), job execution, and every other GC function. Note It is the minimum network performance requirements between OMS and OMR hosts that usually dictate how far you can “spread” an installation, not any inherent limitations of Grid Control, which can scale for hundreds of administrators and tens of thousands of targets. Figure 2-1 shows the minimum bandwidth and maximum latency requirements between OMS, OMR, and OMA tiers. And Table 2-1 summarizes these requirements for easy reference. OMR to OMR storage requirements are also given, but typically the OMR host is physically coupled with its storage devices. Both Figure 2-1 and Table 2-1 show that the minimum bandwidth and maximum latency requirements between OMS and OMR hosts are much more stringent (10× or so) than those between OMS and OMA hosts. (Ideally, OMS / OMA latency should be less than 1ms.) Due to the necessarily tight network coupling between OMS and OMR tiers, most sites run at least one OMS host from within the same data center as the OMR host, usually placing one server next to the other (or these days, one blade on top of the other). Corporate Local Area Networks (LANs) typically run Gigabit Ethernet (GigE) or higher, so they are almost always fast enough to meet the 1Gbps minimum bandwidth requirement between OMS and OMR. Table 2-2 lists LAN bandwidths for the most prevalent LANs. OMS / OMR hosts should perform well if they’re on a GigE LAN. But can you meet GC network requirements (particularly between OMS and OMR) when using multiple OMSs geographically separated and connected over a Wide Area Network (WAN)? Companies with computing resources located around the country (or even around the world) often rely on a WAN to connect LANs at remote office locations, and secure that WAN connection over a VPN tunnel. To establish whether the WAN is fast enough to support the OMS locations you are contemplating, let’s look briefly at typical WAN speeds. How do they fare in relation to GC network requirements? 38 Oracle Enterprise Manager 10g Grid Control Implementation Guide Repository storage 1Gbps 10ms* Repository 300Kbps DSL 300ms* 1Gbps 30ms* Agent DMS 300Kbps DSL 300ms* HTML Console *Min. Bandwidth/Max. Latency FIGURE 2-1. Bandwidth and latency specifications between Grid Control tiers WAN bandwidths vary widely by provider, depending upon the underlying device technology. Most U.S. national and international firms run over at least a DS-1 (Tier 1) or DS-3 (Tier 3) WAN. Based on the broadband transmission rates shown in Table 2-3, DS-1 at 1.544Mbps provides substantially more than the required 300Kbps minimum bandwidth that Agent (and Console) connections with the OMS must meet. However, not even DS-3 transmission speeds of 44.736Mbps would be sufficient to support OMS / OMR network bandwidth requirements, which are nearly 23 times higher (at 1Gpbs). These requirements would necessitate an OC-48 (2.488Gbps) or higher WAN connection. Many firms cannot justify the expense of such a high-speed broadband network. Those companies that Communication between GC Components Minimum Bandwidth Required Maximum Latency Allowed Console <-> OMS 300Kbps 300ms OMA <-> OMS 300Kbps 300ms OMS <-> OMR 1Gbps 30ms OMR <-> OMR Storage 1Gbps 10ms table 2-1. Minimum Network Requirements between Grid Control Components Chapter 2: Grid Control Preinstallation LAN Device Bandwidth Fast Ethernet (100base-X) 100Mbps FDDI 100Mbps Gigabit Ethernet (1000base-X) 1Gbps Myrinet 2000 2Gbps Infiniband 1X 2.5Gbps 10 Gigabit Ethernet (10Gbase-X) 10Gbps table 2-2. 39 LAN Bandwidths for Different LAN Devices can justify the expense can only do so for high-bandwidth networks between primary and disaster recovery (DR) sites, and they lease lines from telecommunications giants like Sprint, Cingular, and AT&T for such purposes. IT departments would be hard-pressed to justify vastly increasing their network budgets just to satisfy a single global Grid Control implementation. System Management Products like Grid Control are usually the orphan step-child of IT departments, and, from a budgetary perspective, must often ride on the coattails of other IT expenditures. As for GC latency requirements, WANs almost always meet the 300ms latency requirements for Agent / OMS or Console / OMS communications. Even dial-up modem users over the Internet (WAN) typically experience only 150-200ms latency. However, while corporate LANs invariably meet the 30 millisecond (ms) OMS / OMR maximum latency, most WANs do not. Again, the exception is a company with a leased line between production and disaster recovery (DR) data centers. They may very well meet this 30ms latency requirement, provided their DR facility is close enough to the primary facility. (The speed of light is the limiting factor in meeting a 30ms latency over thousands of miles). WAN Connection Transmission Speed DS-1 (Tier1) 1.544Mbps E-1 2.048Mbps DS-3 (Tier 3) 44.736Mbps OC-3 155.52Mbps OC-12 622.08Mbps OC-48 2.488Gbps OC-192 9.953Gbps table 2-3. Broadband Transmission Rates 40 Oracle Enterprise Manager 10g Grid Control Implementation Guide Can You Run an Active OMS from a DR Location? As explained in Chapter 15, you can configure a standby OMS host at a DR location and leave it down but ready for service to connect to a standby Repository in case of failover. However, you can also run an active OMS (or an Active/Passive OMS) at a DR location, although other DR hosts remain in standby mode. However, on failover you’d need to point this active OMS located at the DR site to the standby Repository. Wait until Chapter 15 to worry about DR and high availability considerations. For now, you just need to evaluate whether your DR and primary data centers are close enough to place an active OMS at each location. So what does this all mean in terms of the fewest number of sites you can get away with for a particular Grid Control environment? Companies running a VPN over a WAN between offices can use a single GC site for nationwide or even worldwide coverage of each environment (which may just be one environment if it monitors both production and nonproduction targets). Agents and Console connections can span large distances to OMS hosts. However, these OMS hosts must be placed geographically close enough to the OMR node(s) to meet the networking requirements of 1 gigabit per second (Gbps) bandwidth and 30ms latency. This usually means placing one or more OMS hosts alongside the OMR node(s), and if an additional OMS host is required for scalability reasons, placing it at a remote location (perhaps at the DR data center) within 2,790 miles of the production location. This distance can be as the crow flies, or as the satellite flies (i.e., it can be from a Low Earth Orbit (LEO) satellite network typically 500 to 1,500 miles from earth). It so happens that 2,790 miles is the exact distance between New York City and Los Angeles (though why you’d want to place a DR data center at either location is beyond me). How did I come up with 2,790 miles? The speed of light is 186,000 miles per second. If the second OMS host were separated from the Repository host by 2,790 miles, it would take at least 15 milliseconds for each packet to be sent between hosts. Taking packet acknowledgment into account, the latency doubles to 30 milliseconds.5 Tip To “string out” your environment, you can geographically separate the first OMS host from the OMR host. You can then place one or more OMS hosts on the other side of the OMR host from the first OMS host. With this background, you are now ready to start sketching a picture of your proposed Grid Control topology. Take out a piece of paper and a pencil, even crayons if you want—they’re usually closer at hand, thanks to my four-year-old daughter. Let’s paint by numbers now. You might not need to do this if your site is small and all servers are located in the same data center. 5 Thanks to Ed Whalen, one of the technical editors of this book, for the general idea behind this calculation. See his excellent book, Oracle Database 10g Linux Administration, The McGraw-Hill Companies, Inc., 2005, Chapter 13. Chapter 2: Grid Control Preinstallation 41 However, if you have a disaster recovery site located elsewhere, you’ll probably want to go through this exercise. 1. The first fork in the road is to decide how many GC environments are required. This depends upon whether you will use separate GC environments to monitor production and nonproduction targets, and upon your corporate IT management structure. ■■ If your company routinely co locates production and nonproduction targets on the same host, then you should probably build one GC environment (consisting of one or more sites as geographically required) to monitor both production and nonproduction targets. ■■ On the flip side, if your company runs independent regional data centers, perhaps across the country or in different countries, and corporate policy dictates that each data center be run independently, you may need one GC environment per data center (if you’re monitoring production and nonproduction targets together) or even two environments per data center (one for production and the other for nonproduction targets). ■■ If your corporate network (intranet) is slow, geographical considerations alone favor building two regional environments, as dictated by network bandwidth constraints. 2. For each GC environment, draw a crude map to indicate where your GC infrastructure and managed targets will geographically reside. Draw a box on this map to represent each data center, and indicate whether it is a primary or DR location. 3. Figure out how many Grid Control sites you need for each environment, given Grid Control network bandwidth and latency requirements. 4. For each environment and site, position OMR and OMS in these data center boxes where you propose to locate the hosts for these GC components. You need at least one OMR and can have multiple OMSs. Draw a line between each OMS and its associated OMR. Draw OMAs to represent where your target hosts are located and connect them to the OMS(s), either each OMA to a particular OMS (manual load balancing), or each OMA to multiple OMSs (which requires a server load balancer). Now go show that pretty picture to your network administrator to confirm your bandwidth and latency estimates between all components. If the NA makes you sign your drawing in big block letters and hangs it on the lunchroom refrigerator, take out another piece of paper and draw a more realistic picture. Make Required Installation Choices OMR, OMS The installation choices you need to make include answering the following basic questions regarding Grid Control installation options and topologies: ■■ Should you install Grid Control using a new database for the Repository, or should you install Grid Control using an existing database for the Repository? ■■ If you’re using an existing database, should you use a single-instance or RAC Repository database? ■■ If you’re using a RAC database, how many nodes will you need? ■■ How many OMS hosts should you deploy? 42 Oracle Enterprise Manager 10g Grid Control Implementation Guide Tackling these questions requires an understanding of the GC installation options and possible topologies (i.e., defined here narrowly as the host or hosts where GC components will reside). When you kick off the GC installer, the first screen shot asks you to choose from one of four installation types: Installation Type Nickname Enterprise Manager 10g Grid Control Using a New Database Installation Type GC New Database Option Enterprise Manager 10g Grid Control Using an Existing Database Installation Type GC Existing Database Option Additional Management Service Installation Type GC Additional OMS Option Additional Management Agent Installation Type GC Additional OMA Option These installation types have long-winded names and are referenced many times throughout this book, so I use the above nicknames for them. When you launch the Grid Control installer for the first time, you must select either the GC New Database Option or the GC Existing Database Option (after preparing an existing database). These options are mutually exclusive, and you can only select one of these options once for a particular GC environment. The reason is that both options install an OMR either in a new or in an existing database, and only one OMR can exist in a particular GC environment or site. After you select either the first or second option and complete the installation, you can deploy more OMSs by running the GC Additional OMS Option (the third installation type) multiple times, once for each additional OMS host you want to install6. The following table summarizes the GC components that each installation type deploys. Database Created for OMR? OMR Installed? OMS Installed? OMA Installed? GC New Database Option Yes Yes Yes Yes GC Existing Database Option No Yes Yes Yes GC Additional OMS Option No No Yes Yes GC Additional OMA Option No No No Yes Installation Type As the table shows, the first three installation types install an OMS and chain-installed Agent, and only the GC New Database Option creates a database. 6 See Chapter 7 for instructions on the fourth installation type, which is the interactive installation of standalone Agents. Chapter 2: Grid Control Preinstallation 43 The remainder of this section offers a concrete rationale to help you decide what GC installation type to employ, whether to use a RAC Repository, and if so, how many RAC nodes to use, and how many additional OMS hosts to install, if any. Choosing a New or Existing Database for the Repository The GC New and Existing Database Options are mutually exclusive, and you can only select one of these two options one time to build a particular GC site. As I just mentioned, the reason is that both options install a Repository in either a New or Existing Database, and only one Repository can exist in each EM ite. The option you choose determines the GC components installed (at least one component is deployed on the local host), whether you can install them on more than one host, and whether you can use a RAC database for the OMR. Following is a description of these first two options and how they impact component locations on host(s) earmarked for GC installation. GC New Database Option This option lays down all three components (OMR, OMS, and chain-installed OMA) on the same host, creating a new single-instance Oracle 10g Release 1 (10.1.0.4) database to house the OMR. Caution Except for small deployments of fewer than 1,000 targets, Oracle recommends locating the OMR and OMS components on separate, dedicated hosts for performance reasons. The OMR Database should be dedicated to the Repository and not shared with other applications. From a licensing perspective, you can use the OMR Database to host an Oracle Recovery Manager (RMAN) repository 7, but this violates both Grid Control and RMAN best practices. GC Existing Database Option This option installs an OMS and chain-installed Agent on the local host where the installer is run, and installs an OMR on a preexisting database. You can install the OMR in a single-instance or Real Application Clusters (RAC) database. This presents several variations. You can ■■ Install an OMS and chain-installed Agent on one node, and install the OMR in a preexisting database on another node or nodes. This variation has two prongs: ■■ The preexisting database can be a single-instance database. ■■ The preexisting database can be a RAC database. Typically, it is a two-node RAC, but you can start with a single-node RAC and add additional nodes later to scale out. ■■ Install all GC components on the same host as in the first option above8 (i.e., install an OMS and chain-installed OMA on the same host as that running an existing database, and install the OMR in this database). 7 See OracleMetaLink Note 394626.1. The only reason to select this variation is to install a 10.2 or 9i database (which is not recommended) rather than a 10.1.0.4 database installed with the GC New Database Option. 8 44 Oracle Enterprise Manager 10g Grid Control Implementation Guide Tip In Chapter 3, you will learn how to install an OMS on a Repository RAC node. This configuration, supported as of GC 10.2.0.2, is not ideal because the OMR and OMS must compete for resources. However, it may be the only alternative, particularly in a nonproduction environment where server real estate is tight, but where you want to mirror a RAC OMR production environment. The major Grid Control architectural “fork” in the road is whether to choose the GC New Database Option or the GC Existing Database Option. If I were at that fork, and you stopped to ask me which way to go, my answer would have to be conditional. Here are the most succinct directions I could think to give you (I call it fork #1 because there’s one more fork to come): “Fork #1: Take the New Database road if all of the following are true: both the OMR and OMS are certified on your platform, your host meets the combined OMR and OMS hardware requirements, you don’t need high availability for the OMR, and you monitor fewer than 1,000 targets. Otherwise, take the Existing Database road.” This answer is densely packed, so let’s break out a “map” to give you more details on each one of these four conditions. Selecting the GC New Database Option Below is a description of each of the four conditions you should satisfy before choosing the GC New Database Option. ■■ If both the OMR and OMS are certified on your platform The GC New Database Option installs the Repository and OMS on one server, so both components must be certified on whatever operating system that server is running. The OMS can run on most Windows 32-bit platforms and on many UNIX-based 32-bit and 64-bit platforms. However, the OMR database can also run on 64-bit Windows platforms and on additional UNIX 32-bit and 64-bit platforms. See the section “Verify Certification Information” later in the chapter for all OMS-certified platforms. ■■ If your host meets the combined OMR and OMS hardware operating requirements If OMR and OMR components reside on the same host, which is applicable only for small deployments of fewer than 1,000 targets, the combined hardware requirements are two 3GHz CPUs, 8GB RAM, 50GB total disk space, 1.85GB temporary disk space, and 4.5GB swap space. If your host meets these combined hardware requirements, read on about using the GC New Database Option. (Otherwise, you must use the GC Existing Database Option.) ■■ If you don’t need high availability for the OMR If you select the GC New Database Option, it creates a new single-instance Repository. If you need high availability (HA) for the OMR first install a RAC Database or configure the OMR for use with Cold Failover Cluster, which both require selecting the GC Existing Database Option. (You could use the GC New Database Option to install a single-instance database, then RAC it, but it is easier to pre install a RAC database off-the-bat and use the GC Existing Database Chapter 2: Grid Control Preinstallation 45 Option.) If you only need data availability, called cold failover—not high availability— for the Repository, install a single-instance database with the GC New Database Option, and then build a physical standby database for the Repository later using Oracle Data Guard, which also supplies data protection, disaster recovery, and other benefits (see “OMR Database DR Recommendations” in Chapter 15). ■■ If you will monitor fewer than 1,000 targets Whereas the previous point addresses the need for OMR high availability, this point speaks to OMR capacity. Generally speaking, a single-instance Repository, installed using the GC New Database Option, should be able to manage fewer than 1,000 targets, defined as a “small” GC site. A multinode RAC Repository, which requires selecting the GC Existing Database Option, will provide the immediate scalability to monitor more targets. (See the next section for how many RAC nodes you’ll need.) Remember that targets are not just monitored hosts, but all Oracle and non-Oracle products installed on these hosts, such as database instances, listeners, Application Servers, E-Mail servers, and Net App Filers. There are over 100 different target types, and each instance of a particular target type counts as a monitored target. If your site does not meet all four of these conditions for using the GC New Database Option, plan to use the GC Existing Database Option, described next. Caution iSQL*Plus access in Grid Control requires running iSQL*Plus server from any database home. The OMR database home is the logical place for such a home, but the GC New Database Option doesn’t install iSQL*Plus, and you can’t install it afterward, either by rerunning the GC installer or the 10.1 database installer. However, you can install iSQL*Plus in the OMR database home via the GC Existing Database Option. This should not be a major factor in deciding which GC installation type to choose, as you can configure any target database server containing iSQL*Plus to provide iSQL*Plus access in the Console to all target databases. Selecting the GC Existing Database Option While the GC New Database Option installs the Repository in a new single-instance database on the same host as the OMS, you use the GC Existing Database Option to preinstall either a singleinstance or RAC Repository on a different host than the OMS host.9 The only major decision you need to make when using the GC Existing Database Option is whether to use a single-instance or RAC Repository.10 Remember: you’ve already taken the Existing Database road, and now you come to another fork. If, a la Twilight Zone, I magically beat you to this new fork in the road before you 9 You can use the GC Existing Database Option to install the Repository on the same host as the OMS, but this would only be used to preinstall Database version 10.2 or 9.2 (which is not recommended), rather than install Database version 10.1 with the GC New Database Option. 10 Again, please consult Chapter 15, if you’re interested in using an Active/Passive configuration for multiple Repository or OMS nodes. 46 Oracle Enterprise Manager 10g Grid Control Implementation Guide had a chance to drive there, and you were so amazed that you stopped again for directions, I would tell you this: “Fork #2: Howdy. Hope you know you’re on to the GC Existing Database road right now. At this fork, you can either take the single-instance database road or the RAC database road. If you do need high availability for the OMR, or if you will monitor more than 1,000 targets, take the RAC database road.” Otherwise, you can take the single-instance database road.” Finally, how many RAC nodes are required? That’s the easy part. No matter what the GC deployment size is, you shouldn’t need more than a two-node RAC cluster to satisfy requirements for OMR high availability and GC site size. The following table shows this and also quantifies the number of targets that constitute a small, medium, or large GC site. Size of GC Site Number of OMR Nodes (SI or RAC) Small <1,000 targets 1-node, single-instance (or single-node RAC to scale out later) Medium 1,000 to 10,000 targets 2-node, RAC Large >10,000 targets 2-node, RAC Remember that Oracle does not support customers running the OMS on any RAC Repository node until 10.2.0.2, and it’s not recommended in any case, except perhaps for nonproduction sites. Selecting the GC Additional OMS Option After your initial Grid Control installation using either the GC New Database or GC Existing Database Options, you can install additional OMSs on separate hosts by running the GC Additional OMS Option. You must decide how many additional OMS hosts to deploy, if any, and where to locate these hosts geographically. Both decisions depend on the size of your GC site and on high availability (HA) requirements. I’ll use the same definition for a small, medium, and large GC site in establishing sizing requirements. The following table indicates that the total number of OMS hosts needed is as easy as one, two, three (unless you need HA for the OMS component): Size of GC Site Number of OMS Hosts Small <1,000 targets 1 (2 if HA required) Medium 1,000 to 10,000 targets 2 Large >10,000 targets 3 Chapter 2: Grid Control Preinstallation 47 I will defer comprehensive coverage of Grid Control HA considerations until Chapter 15. It should suffice for now to know that providing HA for an OMS in an Active/Active configuration (where both OMS hosts operate independently and concurrently) means installing an additional OMS, even for a small GC site by selecting the GC Additional OMS Option. If you’re considering configuring the OMS in an Active/Passive Cold Failover Cluster (CFC), you must follow a special installation procedure where you actually install both the initial and second OMS concurrently. (Again, see Chapter 15 for details.) If you’re not sure what distinguishes an OMS Active/Active from an Active/Passive configuration, the short answer is that an Active/Passive configuration requires third-party cluster software and shared storage, most commonly NFS. Unless you possess this hardware and storage, you will be using an Active/Active OMS configuration (a best practice anyway, particularly when configured with a Server Load Balancer). There you have it. At this point, hopefully you’ve earmarked whether you’re installing a production or nonproduction GC environment, how many servers you need for the GC installation, and which installation options and component topology you will select. Now, let’s learn how to use the prerequisite checker to determine whether your chosen environment meets certain platform-specific requirements. Then, we’ll cover all network configuration, hardware, and software requirements in detail. Network Configuration Steps The network configuration steps for Grid Control involve setting up host name resolution, adhering to certain host naming rules and constraints, using a static IP address for all GC hosts, and confirming connectivity, most importantly between GC hosts. If you abide by the rules described below, you will avoid possible problems at installation time, and even more painfully, down the road. While it is not very difficult to deinstall and reinstall GC or any of its components after a failed installation, the cost of doing so rises exponentially once you start using and relying upon the product for your monitoring solution. You may be questioning why I place GC network configuration steps ahead of all other requirements. The main reason for getting these network steps out of the way now is that it may take some time for your network administrator to make the network changes, such as fully qualifying the host name or ensuring that it is registered properly in the Domain Name System (DNS). Your company may have host name naming standards that conflict with those presented here, and these conflicts may take time to resolve. At larger corporations, even the simplest network changes may take considerable time to approve and implement. One of those network changes to request from your network administrator is to open any necessary firewall ports between hosts to contain GC components. The easiest way to ensure installation success is to first disable Internet access on all GC hosts (if necessary for security), then open all ports on firewalls between these hosts. You can re secure firewalls after securing traffic between GC components (see Chapter 16). If you disable outgoing Internet access on the OMS before installation, afterward you’ll need to re-enable such access so the OMS can search for and download patches. For information on specific firewall and proxy server configuration requirements, see Chapter 6 of Oracle Enterprise Manager Advanced Configuration 10g Release 4 (10.2.0.4.0). 48 Oracle Enterprise Manager 10g Grid Control Implementation Guide Take Advantage of the Standalone Prereqchecker The Prereqchecker (prerequisite checker) is a routine that automatically validates whether a GC site meets certain (but not all) platform-specific hardware and software prerequisites for OMS and standalone OMA hosts based on the installation type chosen. The GC installer automatically runs the Prereqchecker, or you can run the Prereqchecker in standalone mode. When you run the GC installer and choose an installation type on the first screen, the appropriate Prereqchecker routines automatically run and report their results on the Product-Specific Prerequisite Checks screen. You invoke the Prereqchecker in standalone mode by running the GC installer with the appropriate options, including one to denote the desired installation type. Tip Although the Grid Control installer runs the Prereqchecker for you, I recommend running it in standalone mode. It’s easier to let the Prereqchecker verify requirements and report all results, rather than your having to resolve unmet requirements on the fly during an installation. Use the standalone Prereqchecker in one of the following ways as part of your strategy to confirm all preinstallation requirements: ■■ Start with the standalone Prereqchecker to get an exception report, so to speak, of platform-specific shortcomings, including hardware resource issues, missing packages, insufficient kernel parameters, and the like. Manually redress these shortcomings, and rerun the standalone Prereqchecker to verify that you took care of them. Then using this book as a guide, manually verify your site’s compliance with the remaining prerequisites that the Prereqchecker doesn’t evaluate. ■■ Manually certify that your site meets all prerequisites; then use the standalone Prereqchecker to double-check that you’ve satisfied at least the subset of requirements the Prereqchecker verifies. I personally prefer this method, given that the Prereqchecker does not comprehensively check all GC installation prerequisites and that it’s buggy.11 If you choose the first option, run the standalone Prereqchecker now according to the instructions in Appendix B (online at www.oraclepressbooks.com) under the section “How to Run the Prereqchecker in Standalone Mode.” If you select the second option, wait to run the standalone Prereqchecker until you’ve come to the end of this chapter. (I’ll remind you then.) Also, see “The Prereqchecker” section in Appendix B for a list of the checks performed for each GC installation type (see Table B-1; for now ignore the Additional OMA column), its bugs and quirks, and for sample Prereqchecker output. 11 11 See online Appendix B, “Prereqchecker Bugs and Quirks,” for a complete list. Chapter 2: Grid Control Preinstallation 49 Set Up Host Name Resolution OMR, OMS, OMA The Grid Control installer will fail if you do not first configure name resolution12 on hosts where OMR, OMS, or standalone OMA components are installed. You can configure host names to resolve through the local hosts file on each server, through the Domain Name System (DNS), or both. Note In this section and throughout the book, the term “local hosts file” or “hosts file” is used to signify /etc/hosts on UNIX and C:\WINDOWS\ system32\drivers\etc\hosts on Windows. On UNIX, to determine the order of priority for name resolution, check the “hosts” entry in the /etc/nsswitch.conf file as follows: grep -i hosts /etc/nsswitch.conf The hosts file takes precedence over DNS if the output of this command contains “files” before “dns”, similar to the following: hosts: files [NOTFOUND=continue UNAVAIL=continue] dns Many sites configure their servers with such hosts file priority to minimize network overhead. Conversely, DNS takes precedence over the hosts file if the output from the command above contains “dns” before “files.” ■■ If the hosts file is configured to take precedence (or is used as DNS backup), verify that it contains at least two entries: one for the localhost loopback address and the other for the IP address to hostname mapping. For example, the hosts file must contain at least the following two lines: 127.0.0.1 localhost localhost. . ■■ The localhost entry is a loopback address, and the first alias for it, “localhost,” should not be fully qualified. ■■ If DNS is set to take precedence (or is backup to the hosts file), ensure that you configure both forward and reverse nslookup for each host name running a GC component (see the section “Fully Qualify the DNS Lookup” later in the chapter). Fully Qualify Host Name References OMR, OMS OMR and OMS host names (and target hosts running standalone Agents, if possible) should be consistently fully qualified wherever they are referenced. A fully qualified host name contains the short host name and domain name (such as omshost.mycorp.com) as opposed to just the short host name (omshost). Host names are populated in GC configuration files, and using a 12 Name resolution refers to finding an IP address that corresponds to a given host name. 50 Oracle Enterprise Manager 10g Grid Control Implementation Guide mixture of fully qualified and not fully qualified host names can cause unexpected results, such as GC installation errors, and GC Console or Application Server Console browser pages not found. Not using the fully qualified host name can also cause confusion in that each host across your enterprise may not be uniquely identified. An example where using the short name may be confusing (at best) is when GC manages two target hosts with the same short name (say, host1) in two different countries (such as the United States and the United Kingdom), each with a distinct domain name, (for example, us.mycorp.com and uk.mycorp.com). Grid Control assigns unique names to each target in the Agent’s targets.xml file, but you are only inviting trouble that these hosts will somehow be mixed up. Note Throughout this book, fully qualified host names are used in all explanations and examples. The main places where host name references ought to be fully qualified are as follows (perform all checks on the local host for itself): ■■ In the hostname command output ■■ In the local hosts file ■■ In the DNS Let’s look at each of these host name settings in particular. Tip If your company’s network policy is not to use the fully qualified host name everywhere, at least try to be consistent in not using it anywhere (i.e., use the short host name). This may not be possible, though, as a DNS usually returns the fully qualified host name, whereas network policy or application requirements may dictate using a short name in the hostname output and/or as the first alias in the local hosts file. The best advice I can give you is to lobby to change this network policy, which is inherently inconsistent. Fully Qualify the hostname Command Many UNIX networking programs use the hostname command to identify the machine. Verify that the hostname command returns the fully qualified host name, as follows: hostname . On UNIX only, verify that the domain name has not been set dynamically by executing the following command:13 domainname 13 This command displays the name of the Network Information Service (NIS) domain. NIS is Sun Microsystems’ “Yellow Pages” (YP) client-server directory service protocol for distributing system configuration data such as host names between computers on a computer network. Source: Wikipedia. Chapter 2: Grid Control Preinstallation 51 Verify that no results are returned—the hostname command should statically supply both host name and domain name. If the hostname output is not fully qualified for the OMS or OMR hosts, make this change permanent so that it persists across reboot, provided your company network policy allows it. You must have root privileges on the server to change a host name. Temporarily changing the hostname is also helpful when installing GC on a server that requires reboot for the host name change to take effect, but that can’t be rebooted at the time. You can temporarily change a UNIX hostname by issuing the following command as root: hostname . Do not temporarily change the host name unless you plan to permanently change it by rebooting at the earliest opportunity. A temporary host name change would revert back to the short name after reboot, which would be out of sync with the host name stored in the GC configuration files. Fully Qualify the First Alias in a hosts File As already mentioned, if a server uses a local hosts file for name resolution (even as a backup method behind DNS), this file should include at least the following two lines: 127.0.0.1 localhost localhost. . As for the host name, the first alias (after the IP address) should be fully qualified; not doing so can cause your GC installation to fail when the Web Cache Configuration Assistant runs.14 For example, on a UNIX host named omshost.mycorp.com, the following two lines should be listed in /etc/hosts:15 127.0.0.1 localhost localhost.mycorp.com 138.2.4.84 omshost.mycorp.com omshost Caution When installing Grid Control on SUSE Linux (SLES9), Bug 3843195 causes the bundled Oracle Application Server 10g installation to throw a JAVA error. The workaround is to remove or comment out all IPv6 entries of the following form in /etc/hosts: #::1 localhost ipv6-localhost ipv6-loopback #fe00::0 ipv6-localnet Fully Qualify the DNS Lookup If a GC host relies upon DNS for name resolution, even as a backup method to a local hosts file, a forward or reverse name server lookup (nslookup) of the short name or fully qualified host name 14 OracleMetaLink Note 351841.1. For RAC database nodes, whether for a target or a Repository, in addition to the public IP address entry, you must also include entries for private and VIP addresses. See the Oracle Clusterware documentation for details. 15 52 Oracle Enterprise Manager 10g Grid Control Implementation Guide should return the fully qualified host name. Try the following commands on each GC host, substituting the host name on which you are executing the commands: nslookup . Following is the command output from a forward and reverse nslookup of the host name omshost.mycorp.com (input is in bold): $ nslookup > omshost.mycorp.com Server: 138.2.203.16 Address: 138.2.203.16#53 Name: omshost.mycorp.com Address: 138.2.5.85 > omshost Server: 138.2.203.16 Address: 138.2.203.16#53 Name: omshost.mycorp.com Address: 138.2.4.84 > 138.2.4.84 Server: 138.2.203.16 Address: 138.2.203.16#53 84.4.2.138.in-addr.arpa name = omshost.mycorp.com. Host Name Constraints OMR, OMS, OMA The host names where GC components are installed must adhere to certain naming rules that vary depending upon the component: ■■ OMR, OMS, and OMA host names cannot be localhost, localhost.localdomain, or an IP address, but must be valid host names16. A valid host name is one listed in the local hosts file, or for an OMR or OMA host, a virtual host name as discussed next. ■■ You can use a virtual host name for an OMA or OMR host, but not for an OMS host. A virtual host name is anything other than the physical host name of the server, as indicated by the hostname command output. To use a virtual name, specify the ORACLE_HOSTNAME argument when executing the Grid Control Oracle Universal Installer (OUI). The OUI executable name is different on UNIX than on Windows. On UNIX the OUI syntax to use a virtual host name is: ./runInstaller ORACLE_HOSTNAME= On Windows the syntax is: setup.exe ORACLE_HOSTNAME= 16 The Prereqchecker confirms this requirement. In fact, it’s the only network-related prerequisite check performed. Chapter 2: Grid Control Preinstallation 53 ■■ An OMA or OMR host can have multiple aliases registered with the DNS under a single IP address. If this is the case, set the ORACLE_HOSTNAME argument to the alias you want to use as the host name. You can still set ORACLE_HOSTNAME for the OMS installation, as long as the host name defined is a physical host name, not a virtual host name. ■■ An OMR, OMS, or OMA host can be multihomed17, meaning that it can contain multiple network (NIC) cards, each with its own IP address and host name. If you don’t specify ORACLE_HOSTNAME as an argument to an OMS or standalone OMA installation using the OUI on a host with multiple NIC cards, then the OUI assigns the first multihomed entry in the hosts file as the host name. Use Static IP Addresses OMR, OMS The hosts on which you are installing the OMS and OMR must have unique static IP addresses, not Dynamic Host Configuration Protocol (DHCP) assigned addresses. For hosts running standalone Agents, unique static IP addresses are recommended but not required.18 Connectivity Checks OMR, OMS, OMA Perform the following tests to test connectivity to the SMTP Server, the Internet, and to a Proxy Server (if used). Also check connectivity to OracleMetaLink and check the account credentials Grid Control will use to download patches and patch information. SMTP Server Connectivity OMS, OMA The GC installer prompts you to enter an SMTP Server name and e-mail address where the OMS is to send alert notifications for the default GC administrator called SYSMAN. It is optional to enter this information during the installation but highly recommended because it immediately enables notifications for out-of-box alert thresholds. Test SMTP Server If you enter an SMTP server, the installer must confirm this server is reachable from the OMS host before you can advance to the next screen. If the SMTP server is not reachable, you must uncheck the box to enable e-mail notifications. For this reason, it is wise to confirm before starting the installer that all OMS hosts can reach this SMTP server. 17 An OMR host can also be multihomed, but the ORACLE_HOSTNAME argument is not used to specify the OMR host name (the first multihomed entry in the hosts file is used). 18 You can run Grid Control on your personal computer (PC) if it has a static IP address. Given that most personal computers run on DHCP-assigned networks, an alternative is to run the Grid Control Vmware bubble. Unfortunately, it is only available for download from an internal Oracle site, the Oracle Fusion Factory at http://ff. us.oracle.com/vmware. Your local Oracle sales rep or on-site Oracle Consultant would be more than happy to demo this bubble (and any others you may be interested in), and perhaps even leave it for you to take it out on an unaccompanied test drive. Even the bubble needs a static IP address, which you provide by adding the Vmware image IP address to the hosts file on your PC. This bubble runs on Red Hat Linux 3.0 and requires the following minimum hardware: a Pentium IV 2.6Ghz or Centrino 1.4Ghz processor, 2GB of RAM, and 24GB of free disk space. 54 Oracle Enterprise Manager 10g Grid Control Implementation Guide In addition, it is important that the Monitoring Agent (the Agent chain-installed with the initial OMS) and any additional Agents desired be able to send an out-of-band (OOB) notification through this SMTP server. An OOB notification is an SOS call sent directly through an SMTP server (not through an OMS) to a designated e-mail address to relay that no OMS hosts are online to receive Agent alerts or data uploads. You will configure the OOB notifications themselves in Chapter 11 in the section “Enable Notification if Grid Control Goes Down.” For now, just make sure that the Monitoring Agent, and, preferably, at least one target host per subnet, can ping the SMTP Server: ping -c 4 Test Notification E-mail Address Next, from each OMS host, make sure the SMTP server can relay notifications to the e-mail address you want to associate with the SYSMAN account (and to any other e-mail addresses where you want to receive notifications). This is also important to check in advance because the GC installer does not confirm receipt to the e-mail address you supply. The e-mail address can be a generic administrator e-mail account, to which you can assign one or more administrators, or a distribution list, as you see fit. Send a test e-mail to this address via a telnet connection to the SMTP server on port 25 from the command line. Following is the output from such a telnet session on the host omshost. In this session, we send a test e-mail from sender EnterpriseManager@oracle.com, which must be an e-mail account that your SMTP server recognizes, to the same e-mail address. Input commands are shown in bold: $ telnet mail.mycorp.com 25 Trying 138.1.161.138... Connected to bigip-mail.mycorp.com (138.1.161.138). Escape character is '^]'. 220 rgmsgw02.mycorp.com ESMTP Mycorp Corporation Secure SMTP Gateway Switch-3.1.7/Switch-3.1.7 - Ready at Wed, 5 Apr 2006 11:43:43 -0600 Unauthorized Usage Prohibited. helo 250 rgmsgw02.mycorp.com Hello omshost.mycorp.com [138.2.4.61], pleased to meet you mail from: EnterpriseManager@mycorp.com 250 2.1.0 EnterpriseManager@mycorp.com... Sender ok rcpt to: EnterpriseManager@mycorp.com 250 2.1.5 EnterpriseManager@mycorp.com... Recipient ok data 354 Enter mail, end with “.” on a line by itself Subject: SMTP Test Hello this is an SMTP test for Grid Control . 250 2.0.0 k35HhhOm026568 Message accepted for delivery quit 221 2.0.0 rgmsgw02.mycorp.com closing connection Connection closed by foreign host. Confirm that you receive this test e-mail at EnterpriseManager@mycorp.com in a timely manner. Using this telnet method, also confirm that Monitoring Agent OOB notifications will succeed by sending a test e-mail from each Monitoring Agent to the e-mail address you plan to designate for OOB notifications. Chapter 2: Grid Control Preinstallation 55 OMS MetaLink Connectivity OMS The OMS must be able to connect to OracleMetaLink at http://oracle.com/support/metalink/ index.html in order to locate and download patches. Open a browser session from the OMS and confirm Internet access to this URL. If the OMS must go through a proxy server to reach the Internet, this confirms that the browser’s proxy server information is correct. Ensure that you provide this same proxy server information during the installation. Also verify the OracleMetaLink credentials that the OMS will use for patch-related functionality. I recommend creating a new OracleMetaLink account name for Grid Control to use exclusively, such as “mycorp_OEM.” If you cannot install a browser on the OMS to check OracleMetaLink connectivity and credentials, at least check that the OMS can access OracleMetaLink by pinging it: ping -c 4 updates.oracle.com Be aware that this ping will fail if your network administrator disabled ping access from inside the firewall to external sites. Connectivity Between All GC Hosts OMR, OMS, OMA The OMS needs to be able to communicate in both directions with the OMR host via JDBC, and with OMA hosts via HTTP and HTTPS. If a firewall separates the OMS from the OMR or, more likely, if a firewall separates the OMS from the OMA, check that your company’s proxy server is configured as required to allow communications between these GC components. First, from the OMS host, ping the proxy server between OMS and OMR, if one exists: ping -c 4 Second, from the OMS, ping the OMR and the OMA to confirm that they are reachable. Also ping in the reverse direction from the OMR to the OMS and from the OMA to the OMS. Specify fully qualified host names when executing all ping commands. Tip A check (“✔”) symbol next to a subheading either under hardware and software requirements below or in Appendix B (online) indicates that the Prereqchecker confirms this prerequisite for you. Table B-1 in Appendix B summarizes all Prereqchecker actions. Hardware Requirements Requirements for installing Grid Control versus operating Grid Control in production are vastly different. This section covers both, in that order. The installer Prereqchecker confirms that your system possesses the minimum resources just to install Grid Control (at least in my opinion). These resources include hardware requirements such as CPU, RAM, and disk space. Whether the hardware has enough horsepower to go live with a GC production site supporting all your managed targets for at least the short term … well, that’s another story. 56 Oracle Enterprise Manager 10g Grid Control Implementation Guide First, let’s spell out the minimum hardware requirements to get through a GC installation. This is useful to know if you don’t yet have the hardware resources to put Grid Control into production but want to get started and test-drive its functionality. Hardware Installation Requirements ✔ OMR, OMS, OMA If your production Grid Control environment requires high availability HA (addressed in detail in Chapter 15), then you need to design and build in fault-tolerant storage up front for all GC framework components—OMR database, OMR software, OMS software, and OMA software. The topic of fault-tolerant storage falls outside the scope of this book. However, you’ll find an excellent treatment on best practices for configuring HA storage in Section 2.1 of Oracle Database High Availability Best Practices 10g Release 2 (10.2). This section identifies the following six elements of a fault-tolerant database storage subsystem, but the last three of these elements apply equally to storage for all GC framework components, as indicated in the table below: Element of Fault-Tolerant Storage Applies To Evaluate database performance requirements and storage performance capabilities OMR Database Use Automatic Storage Management (ASM) to manage database files OMR Database Use a simple disk and disk group configuration OMR Database Use disk multipathing software to protect from path failure All GC Framework Components Use redundancy to protect from disk failure All GC Framework Components Consider HARD-compliant storage All GC Framework Components Table 2-4 lists the hardware installation requirements on a per-host basis when installing Grid Control using the first three of four GC GUI installation types (all except Additional Management Agent).19 Remember, these are only installation minimums, not operating requirements (listed next). All requirements are per host. The GC Existing Database Option is represented in two rows of the table. The first row is for the host where the OMS and chaininstalled OMA are installed, and the second row is for the host where the OMR is installed. 19 The requirements are specified as parameters in the file oraparam.ini located in the install directory at the top level of the EM software distribution. The total disk space installation footprint is broken out by OS, and is for all component Oracle homes installed for the indicated installation type. The installation prerequisite checks verify all requirements except those for the OMR host as specified in the row GC Existing Database Option (OMR Host), as this host is different from the installation host. Chapter 2: Physical RAM Total Disk Space Installation Type (Component Host) AIX Grid Control Preinstallation Temporary Disk Space 57 Swap Space HP-UX Linux Solaris AIX, HPLinux, UX, Linux, Solaris, AIX, All Windows Platforms Windows HP-UX Windows Solaris GC New Database Option20 (OMR/ OMS/OMA Host) 9.0GB 9.0GB 4.5GB 4.2GB 512MB 250MB 1.3GB 250MB 500MB GC Existing Database Option (OMS/OMA Host) 5.0GB 5.0GB 2.5GB 2.5GB 512MB 250MB 1.3GB 250MB 500MB GC Existing Database Option (OMR Host) 4.0GB 4.0GB 2.0GB 1.7GB 960MB 250MB 0GB 250MB 500MB Additional OMS (OMS/OMA Host)21 2.0GB 2.75GB 2.0GB 2.5GB 512MB 250MB 1.3GB 250MB 500MB table 2-4. Type Hardware Installation Requirements per OMR/OMS Host(s) Broken Down by Installation Hardware Operating Requirements OMR, OMS Following is a summary of the hardware operating requirements for each OMR host (Table 2-5) 2021 and OMS host (Table 2-6), including number of CPUs,22 RAM, total disk space, temporary disk space, and swap space, needed to sustain the indicated GC site sizes. Because the units of these specifications are on a per-host basis, the number of OMR and OMS hosts listed in earlier tables is listed again for the sake of completeness in Tables 2-5 and 2-6. Note Again, these requirements assume the OMR and OMS components reside on different hosts, which Oracle recommends for performance reasons. However, if these components reside on the same host, which is acceptable for small deployments of fewer than 1,000 targets, the combined OMR/OMS requirements are two 3GHz CPUs, 8GB RAM, 50GB total disk space, 1.85GB temporary disk space, and 4.5GB swap space. 20 These requirements also apply to the GC Existing Database Option if the OMR and OMS are installed on the same host. 21 The Prereqchecker does not verify some of these installation requirements, including physical RAM, temporary space, and swap space. 22 Expect diminishing performance returns from OMR or OMS hosts running multicore microprocessors. In other words, don’t expect 2× the performance from a dual-core device over a single-core processor. 58 Oracle Enterprise Manager 10g Grid Control Implementation Guide Number of OMR Hosts Number of CPUs (speed) Physical RAM Total Disk Space Temporary Disk Space Swap Space (UNIX Only) Small <1,000 targets 1-node, SingleInstance 2 (3GHz) 4GB 40.0GB 1.6GB 4.0GB Medium 1,000 to 10,000 targets 2-node, RAC 4 (3GHz) 8GB 80.0GB 1.6GB 8.0GB Large >10,000 targets 2-node, RAC 8 (3GHz) 12GB 300.0GB 1.6GB 12.0GB Deployment Size table 2-5. Hardware Operating Requirements per Host Dedicated to the OMR Deployment Size Number of OMS Hosts Number of CPUs (speed) Physical RAM Total Disk Space Temporary Disk Space Swap Space (UNIX Only) Small <1,000 targets 1 2 (3GHz) 4.0GB 10.0GB 250MB 500MB Medium 1,000 to 10,000 targets 2 4 (3GHz) 4.0GB 10.0GB 250MB 500MB Large >10,000 targets 3 4 (3GHz) 4.0GB 20.0GB 250MB 500MB table 2-6. Hardware Operating Requirements per Host Dedicated to the OMS The range of values correlates with the deployment size (i.e., number of targets) and significantly exceeds the minimum installation hardware requirements above.23 Notice that as Grid Control scales upward, each OMR node needs proportionally more CPU and RAM than each OMS. This is partially explained by the fact that more OMS than OMR nodes are needed as the deployment size increases. The commands to check CPU, RAM, swap space, disk space, and temporary disk space requirements for all supported operating systems are listed in Table 2-7. 23 The disk space allotments for both OMR and OMS are generous compared with those specified in the Oracle EM documentation, and allow for growth over a site’s lifetime. The OMR disk space is sized to handle expected log and trace files, growth in the OMR database tables as well as in an external Change Manager schema, and a Flash Recovery Area large enough to hold two days of on-disk RMAN backups (and archive logs), a block change tracking file, and 24 hours of flashback data. (Chapters 3 and 4 detail OMR database configuration.) The OMS disk space is double that recommended in the EM documentation based on real-world experience. It allows for centralized storage of patches, preferably on a shared drive accessible by all OMS hosts. This allows administrators to deploy patches when logged into the Console via any OMS. Chapter 2: Grid Control Preinstallation 59 AIX HP-UX Linux Solaris Windows Number (speed) of CPUs24 lscfg25 | grep 'proc[0-9]' | awk 'END {print NR}' top ioscan -C processor cat /proc/ cpuinfo prtconf -v | head Right-click Computer Name, select Properties, and then select the General tab. You can also get the number of CPUs from environment variable NUMBER_OF_ PROCESSORS. Physical RAM bootinfo -r lsattr -EHl such as mem0 grep "Physical:" /var/adm/syslog/ syslog.log grep MemTotal /proc/meminfo /usr/sbin/ prtconf | grep "Memory size" Right-click Computer Name, select Properties, and then select the General tab. Swap Space swap -s lsps -a (for paging info) /usr/sbin/ swapinfo -a grep SwapTotal /proc/meminfo /usr/sbin/ swap -s Right-click Computer Name, select Properties, and then select the Advanced tab. Under Performance, click Settings, which brings up Performance Options. On the Advanced tab under Virtual memory, Total paging file size for all drives is listed. Total Disk Space df bdf df df Right-click the local disk where GC software will be installed. Temporary Disk Space26 df /tmp bdf /tmp df /tmp df /tmp Defaults to C:\Documents and Settings\ \Local Settings\Temp. Right-click drive and select Properties. table 2-7. Commands to Check Hardware Requirements on All Certified Platforms Because of the minimum requirement of 4GB of RAM for the OMR, I highly recommend 242526 running the OMR on a 64-bit platform. On a host dedicated to an Oracle database, such as the GC Management Repository, most of the RAM is usually assigned to the Oracle System Global Area (SGA). So even the smallest of GC sites running an OMR on a host with 4GB of RAM requires more than 2GB SGA. On all 32-bit platforms, attaining an SGA above 1.7GB involves implementing the Very Large Memory (VLM) option for that platform, as discussed in Chapter 17 (see PlatformSpecific Performance Recommendations). The VLM option introduces complications of its own on all 32-bit platforms. By contrast, if you run the OMR on a 64-bit platform that natively supports large SGAs, you avoid these VLM complications. (Linux 64-bit platforms require implementing HugePages, which is technically a VLM option, but measurably improves performance.) Also, memory performance on 64-bit platforms is far superior to that of 32-bit platforms running VLM, not to mention the superior performance that 64-bit platforms afford in every other regard. 24 For existing databases, you can also look at the initialization parameter CPU_COUNT that Oracle automatically determines. However, this parameter is not always accurate for several reasons. Dual-CPUs may not be accounted for. Also, beware of Bug 2735470 fixed in Oracle 10.1.0.1 that causes Oracle to ignore any CPUs you add after installing the Oracle database. 25 lscfg is included in the AIX package called Monitor. 26 You can supersede the default temporary directory using environment variables TEMP and TMP (set both to be sure). 60 Oracle Enterprise Manager 10g Grid Control Implementation Guide Software Requirements This section details the following software requirements27 for OMR, OMS, and standalone OMA hosts: ■■ Verify certification information ■■ Create required OS groups and users ■■ Create required directories ■■ Stop database listeners using port 1521 ■■ Synchronize OS timestamps/time zones ■■ Confirm platform-specific software requirements All supported GC platforms must meet these generic prerequisites (all requirements except platform-specific ones). Such requirements include creating the requisite OS groups, users, and directories that the GC installer needs. Note If you are using a preexisting database to house the OMR, you have probably already satisfied the OMR host generic software requirements, entitled Create Required OS Users and Groups and Create Required Directories, as they are also Oracle Database preinstallation steps. In addition to generic requirements, the section “Confirm Platform-Specific Software Requirements” at the end of this chapter lists unique requirements for each platform and OS version, such as required Solaris 9 patches that apply only to this operating system. Verify Certification Information OMR, OMS, OMA One of the GC software preinstallation steps is to check that all components (OMS, Repository, Agents, Console) and Monitoring Plug-ins are certified on the intended platforms and for applicable O/S, database and browser, and third-party hardware and software versions. To verify that your intended GC environment is supported, consult the “Certification Information” section in Appendix B (online), which also explains the reasons behind, implications of, and recommendations related to the certifications. Appendix B contains certification information for Oracle Enterprise Manager 10g Grid Control Release 3 (10.2.0.3) and Release 4 (10.2.0.4.0), the latest two 10.2 releases available. I’ve limited the data to these two releases because, between them, Grid Control runs on all major operating systems found in today’s data centers. Some certification information 27 I define software requirements much more broadly than does Oracle Enterprise Manager Grid Control Installation and Basic Configuration. In Chapter 1 of that Oracle manual, the section entitled Enterprise Manager Software Requirements limits software to Oracle software that GC depends upon, such as the Oracle database software required for the Repository when choosing the GC Existing Database Option. In this section, software requirements also include software configuration steps and platform-specific software requirements. Chapter 2: Grid Control Preinstallation 61 may have changed since the book’s publication. Therefore, double-check everything against the Oracle Enterprise Manager 10g Grid Control Certification Checker, OracleMetaLink Note 412431.1. This note is accessible from the Certify tab on the OracleMetaLink home page and provides an up-to-date matrix of certification data. Create Required OS Groups and Users OMR, OMS, OMA Create the identically named OS users and groups below on all hosts housing GC components, if these users don’t already exist. Follow a consistent naming convention to simplify administration. ■■ Oracle inventory group (oinstall ) ■■ OSDBA group (dba) ■■ OSOPER group (oper). This group is optional. ■■ Oracle software owner (oracle). This owner is also used on Windows platforms. ■■ Unprivileged user (nobody). This user exists on UNIX platforms only. For UNIX platforms in particular, also use the same user ID (uid), group ID (gid), and supplemental group name for the oracle user across all servers where GC components are installed. Hosts containing GC components may require consistent user and group names to one degree or another, as shown in these examples: ■■ Choose the same Agent user and groups that own (or predominantly own) existing Oracle targets to be monitored. Additional configuration is sometimes required for Agents to monitor targets installed by a different user (see Chapter 5). ■■ All RAC Repository nodes require the same Oracle users, user IDs, groups, and group IDs named above.28 ■■ A Repository Data Guard configuration requires or recommends the same Oracle users, user IDs, groups, and group IDs named above on all primary and standby nodes. Create Oracle Inventory Group (oinstall) If you install Grid Control on a UNIX system with no existing Oracle products,29 the OUI prompts you to create a new Central Inventory to store information about the GC software (see Figure 3-4 in Chapter 3) and any other Oracle software later installed on the host. You must enter the inventory directory path (the standard name to use for a Central Inventory is $ORACLE_BASE/oraInventory) and OS group name (typically named oinstall) with write permission to the inventory directory. The OUI then creates the Central Inventory in the specified location. The OUI also creates a Central Inventory pointer file (called oraInst.loc) in a default location to point to the inventory path. The following table 28 Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (10.2), Chapter 2 (for all UNIX platforms). 29 Technically, the OUI prompts you to create a new Central Inventory if no Central Inventory pointer file oraInst .loc exists on the host. 62 Oracle Enterprise Manager 10g Grid Control Implementation Guide shows the default location of the oraInst.loc file (and the oratab file discussed in the section “Create Oracle Base Directory” later in the chapter) for all UNIX platforms: Platform Location of Configuration Files oraInst.loc and oratab AIX /etc HP-UX /var/opt/oracle Linux /etc Solaris /var/opt/oracle The inventory pointer file consists of two lines following the example format here: inventory_loc=/u01/app/oracle/oraInventory inst_group=oinstall The parameter inventory_loc is set to the specified inventory path, and the parameter inst_ group is set to the specified inventory group owner. First, determine if there is already an Oracle inventory group by checking whether the oraInst .loc file exists in the default location for your platform. If the oraInst.loc file exists, verify that the group name defined by the parameter inst_group is an existing OS group in the /etc/group file, as follows (assuming oinstall is the group name): grep oinstall /etc/group If the Oracle inventory group doesn’t exist, login as root and create it as follows, which varies by platform: ■■ On HP-UX, Linux and Solaris, enter: /usr/sbin/groupadd -g 501 oinstall This example assumes you are creating the oinstall group with group id (gid) 501. Specify the same group name and gid on all hosts containing GC software. ■■ On AIX, enter: smit security Choose the appropriate menu commands to create the oinstall group, and press f10 to exit. On Windows platforms, the Grid Control OUI does not prompt you to create a new Central Inventory, but creates one in the default location %SystemDrive%:Program Files\Oracle\Inventory\. Windows does not use an inventory pointer file, but instead specifies the inventory location in the registry key HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\INST_LOC. Windows creates an OS group called OSDBA to own the Central Inventory. Create OSDBA Group (dba) This step is only necessary if you plan to select the GC New Database Option, which creates a new Repository database. In this case, you must create an OSDBA group if one doesn’t already exist. This group identifies operating system user accounts that have database administrative Chapter 2: Grid Control Preinstallation 63 privileges (the SYSDBA privilege) on the Repository Database. The standard name for this group is dba. Check for the existence of the OSDBA group by doing the following: ■■ If an OSDBA group does not exist, create an OSDBA group using the same command syntax listed above for the Oracle inventory group. ■■ If an OSDBA group already exists, you can use this group after installation to log in to the Repository Database with operating system authentication. On the other hand, if you don’t want the users in the OSDBA group to have the SYSDBA privilege on the Repository Database, create a new OSDBA group now to give a different group of OS users the SYSDBA privilege. Create OSOPER Group (oper) This step is only required if you plan to select the GC New Database Option, which creates a new Repository database. If so, create an OSOPER group only if you need a group of OS users with a limited set of DBA privileges (SYSOPER operator privileges). For most GC environments, an OSOPER group is not required. The default name for this group is oper. If you need an OSDBA group, check for its existence and create it using the same command syntax listed above for the Oracle inventory group. As with the OSDBA group, you can create a new OSOPER group if one already exists to give a different group of OS users (other than the existing OSOPER group) operator privileges for your GC installation. Create Oracle Software Owner (oracle) The GC software owner is typically the user oracle, and will be so throughout this book. This user should have the Oracle inventory (oinstall) group as its primary group and the OSDBA (dba) group as a supplemental (or secondary) group.30 Caution If other Oracle software is already installed on a server, and the Oracle inventory group (oinstall) owns the inventory, then this oinstall group must be the primary group of the OS user installing the GC software. You can create a different user name other than oracle, if one already exists, to assign a different user ownership for your GC installation. Check for the existence of this user as follows: id oracle If the oracle user exists, then the command output will look something like this: uid=200(oracle) gid=500(oinstall) groups=501(dba),502(oper) If the user exists, then decide whether to use the existing user or create another user. To use the existing oracle user, confirm as shown above that the primary group is the Oracle inventory group (oinstall), and that it is a member of the OSDBA group and, optionally, the OSOPER group. 30 Some sites do not use the oinstall group at all, preferring only to use the dba group. This will also work, though it is not a best practice. The primary group is the first group listed for “groups=” when you use the id command, and is specified by the useradd or usermod “-g” option. Supplemental groups are defined by the “-G” option. 64 Oracle Enterprise Manager 10g Grid Control Implementation Guide To create a new Oracle software user, either because it doesn’t exist or because you require a new one, use the following command syntax as root, depending upon the platform: ■■ On HP-UX, Linux, or Solaris, to create an oracle user with the properties shown above (assuming the oinstall and dba groups already exist with the gid’s listed): /usr/sbin/useradd -u 200 -g oinstall -G dba,oper -s /bin/bash -m oracle where ■■ -u is the numerical value of the user’s ID ■■ -g is the group name of the user’s initial login group ■■ -G is a list of supplemental groups of which the user is a member Set the password for the oracle user as follows: passwd oracle ■■ On AIX, enter the following command to create the oracle user: smit security Choose the menu commands to create the oracle user as follows: In the Primary GROUP field, specify the Oracle inventory group, oinstall. In the Group SET field, specify the OSDBA group dba, and, if required, the OSOPER group oper. Press f10 to exit. Then set the oracle user’s password: passwd oracle If the oracle user exists, but its primary group is not oinstall or it is not a member of the OSDBA or OSOPER groups, then enter the following command to modify it (not all usermod options below may be necessary): /usr/sbin/usermod -g oinstall -G dba,oper -u 200 oracle Create Unprivileged User (nobody) The unprivileged user called nobody must exist only if you plan to select the GC New Database Option, which creates a new Repository database. If so, verify that the nobody user exists. The nobody user must own any external jobs (extjob) executable after the installation. To see whether the nobody user exists, enter the following command: id nobody If the nobody user does not exist, then enter the following command to create this user on HP-UX, Linux, or Solaris: useradd nobody On AIX, enter the following command to create this user: smit security Enter the nobody user name, and leave the Primary GROUP and Group SET fields blank. Chapter 2: Grid Control Preinstallation 65 Create Required Directories OMR, OMS, OMA It is best practice that all GC directory structures adhere to the Optimal Flexible Architecture (OFA) guidelines, and that you consistently specify these three directories across all GC hosts. Create or choose in advance paths for three such directories that the GC software must use: ■■ Oracle base directory ■■ Oracle inventory directory ■■ Grid Control parent directory Using the same directory structure on all hosts allows you to easily find the home or inventory, hard-code Agent home directory paths into generic Agent scripts (if absolutely necessary), and document your Agent installations. Here is a brief summary of the tasks to perform in this section. Before running the GC installer, ensure the Oracle base directory exists and has permissions for the oracle user to create the GC Oracle homes below it. Also, decide upon Oracle inventory and GC parent directory specifications, and back up any preexisting Oracle inventory directory. Caution The Grid Control installer and Console cannot process symbolic links or spaces, so don’t use them in directory paths, in configuration files, or in the User Interface. Create the Oracle Base Directory The Oracle base directory, defined by the OS environment variable ORACLE_BASE, is the top-level directory for all Oracle software installations. OFA guidelines recommend the following path: ORACLE_BASE= /app/oracle where is the file system to contain the GC software. This directory specification assumes that the oracle user is the owner of the Oracle software. You can either use the same Oracle base directory for all installations or separate Oracle base directories for each Oracle product. If a different OS user owns each Oracle product on the same server, then you can create a separate Oracle base directory for each user. Hopefully, you are following best practices, dedicating OMS and OMR hosts to your GC infrastructure, with no other products installed on these hosts. In this case, there will be no preexisting Oracle base directory. However, if this is not the case, or you’re not sure whether an Oracle base directory already exists, you can usually determine what it is by any existing Oracle inventory or Oracle home directories. Directory paths that end with the Oracle software owner are likely candidates for existing Oracle base directories. Check both oraInst.loc and oratab files for such directory paths. For example, the inventory_loc value in the oraInst.loc file is usually an Oracle base directory. For the setting used in the earlier section “Create Oracle Inventory Group (oinstall)”, inventory_loc=/u01/app/oracle/oraInventory 66 Oracle Enterprise Manager 10g Grid Control Implementation Guide the parent directory, /u01/app/oracle, is probably an Oracle base directory. Similarly, if you’re using the oracle user to install the GC software, and an oratab file exists and contains the following two lines, prod:/u02/app/oracle/product/10.2.0/db_1:Y *:/u03/app/iasora/product/10.2.0/as10g/:N then /u02/app/oracle would be a viable Oracle base directory to use. Ideally, you should place this directory on a different file system than the operating system, and make sure it has enough free disk space to satisfy Grid Control as specified in the section “Hardware Requirements” earlier in this chapter. Later in this chapter, when you set up the environment for the oracle user to run the installer, you will be reminded to set the ORACLE_BASE environment variable accordingly. To create the Oracle base directory with the correct owner, group, and permissions, enter the following commands (you probably need to be logged in as root): mkdir -p /u01/app/oracle chown -R oracle:oinstall /u01/app/oracle chmod -R 775 /u01/app/oracle Choose the Oracle Inventory Directory If you install Grid Control on a UNIX system with no existing Oracle products, the OUI prompts you to enter the Oracle inventory directory path and creates the inventory in the specified location with the proper owner, group, and permissions. The installer also creates an oraInst.loc file in the default location for your platform to point to the inventory directory (refer back to the earlier section “Create Oracle Inventory Group (oinstall)” for more background). Do the following now to make sure the OUI can perform these steps when you run the GC installer in Chapter 3: 1. If the oraInst.loc file does not exist in the default location for your platform, the GC installer will create this file, provided the GC OS user has write permissions to do so (confirm this now). 2. If the oraInst.loc file exists in the default location: ■■ Ensure root owns this file with 644 permissions, and that inventory_loc points to the desired location for the Grid Control inventory. ■■ Ensure the GC user (oracle) and/or group (oinstall ) own the inventory directory with 664 permissions so that the GC installer can update the inventory. If needed, recursively grant such access to the inventory directory as follows: chown -R oracle:oinstall chmod -R 664 On Windows hosts, you must use a Central Inventory. However, on UNIX systems, you can manage inventories on hosts where GC components, including Agents, are to be installed using one of two approaches: ■■ A Central Inventory ■■ A separate inventory for Grid Control Chapter 2: Grid Control Preinstallation 67 Use a Central Inventory Installing Grid Control (or any Oracle product) into an existing registered Central Inventory is the Oracle best practice. It is also a Grid Control best practice to install the OMS and Repository Database on a dedicated host or hosts. In such a case, no Oracle inventory will exist when installing the OMS or OMR. However, on target hosts where you will deploy Agents, a Central Inventory will probably exist for Oracle products that Grid Control is to manage. Using a Central Inventory to contain both GC software and any other Oracle software on a host (rather than using a separate inventory for GC) greatly reduces inventory administration. For example, after installation, inventory information for GC and target software alike will automatically appear in the Console under the Deployment tab without additional GC configuration required (described in Chapter 11 in the section “Enable Multi-Inventory Support”). One possible downside of using a Central Inventory is that, although it’s a best practice, it can cause deinstallation, upgrade, and patching problems for Oracle products (including GC components) sharing that inventory, but only if you don’t take proper precautions to back up the inventory before running the OUI. Unless you have a good reason to use a separate inventory for GC installations, I would recommend using a Central Inventory. If a Central Inventory exists on a host where you plan to install a GC component (likely just the Agent), do the following to validate and back up the inventory on each of these hosts: 1. Confirm the OUI version, accessibility, and validity of the Central Inventory. Run the following command as the owner of the GC component to be installed: opatch lsinventory ■■ Confirm from the value of the OUI version that the preexisting Central Inventory was created with a version 10.2 Oracle Universal Installer (the same OUI version the GC installer uses). While a preexisting inventory created by a 10.1 installer may work, you’re asking for trouble, based on my experience. Unless you want to upgrade your OUI to 10.2, I’d suggest using a separate inventory for Grid Control in this case. ■■ Verify that the existing Central Inventory is not corrupted and is accessible (i.e., can be run) by the user who owns the GC component to be installed. Confirm that no errors appear in the output from the command above, and that the last output line says “OPatch succeeded.” 2. Prepare to install any GC component on the host by backing up the preexisting Central Inventory and the corresponding oraInst.loc file pointing to it. If the GC installation fails, you can simply restore the backup of the preexisting Central Inventory rather than having to detach GC from it using the OUI. ■■ On Windows, backup Central Inventory at %SystemDrive%:Program Files\Oracle\ Inventory\ to another location. ■■ On UNIX, backup the inventory as follows: cd /.. cp -Rp Use a Separate Inventory for Grid Control Your site may maintain separate inventories for each Oracle product installed on a particular host. In this case, you may prefer to register the GC installation in a new separate inventory in one of the following ways, as illustrated in Figure 2-2. 68 Oracle Enterprise Manager 10g Grid Control Implementation Guide Change inventory_loc value to point to new inventory for GC oraInst.loc /u01/app/oracle/oraInventory inventory_loc=/u01/app/oracle/oraInventory.GC inst_group=oinstall /u01/app/oracle/oraInventory.GC Move existing inventory to make room for new GC inventory oraInst.loc /u01/app/oracle/oraInventory inventory_loc=/u01/app/oracle/oraInventory inst_group=oinstall /u01/app/oracle/oraInventory.old FIGURE 2-2. The two different methods to use separate Oracle Inventories Here is a description of the two methods for moving the inventory: ■■ Change inventory_loc value to point to new inventory for GC Change the inventory_loc value in the inventory pointer file oraInst.loc to point to a new location. In Figure 2-2, we are changing inventory_loc from /u01/app/oracle/oraInventory to /u01/app/oracle/oraInventory .GC. First back up the existing default oraInst.loc file to a meaningful name indicating the corresponding Oracle product already installed (or “pound out” the current entry in the file by entering a “#” as the first character of the line). Then change the inventory_loc value in the default oraInst.loc file to point to a new inventory path for the GC installation. ■■ Move existing inventory to make room for new GC inventory Move the existing inventory directory containing information on existing Oracle products and specified by inventory_loc to an alternate location, and use the existing inventory_loc location for the new GC Central Inventory. In Figure 2-2, we move the inventory directory /u01/app/ oracle/oraInventory to /u01/app/oracle/oraInventory.old. Chapter 2: Grid Control Preinstallation 69 If you decide to use a separate inventory for each product, you will later need to configure Grid Control to discover each inventory located in a nondefault location (other than that pointed to by the default oraInst.loc file). This configuration process is covered in Chapter 11 in the section “Enable Multi-Inventory Support.” Choose the Grid Control Parent Directory For all GC components on all hosts, use the same GC parent directory specification if possible (the directory that the GC installer prompts for that contains all the GC Oracle homes as subdirectories). The parent directory should be a subdirectory of the Oracle base directory. You do not need to pre create this parent directory; the GC installation will create it as long as the oracle user has read-write permissions and the oracle group has read permissions (750 on UNIX) on the lowest existing directory (typically ORACLE_BASE) comprising the parent directory. If you do pre create a parent directory, the Prereqchecker warns that it exists (the installer still continues); it checks that the path doesn’t contain a space; and it performs an Oracle Home Compatibility check that the component oracle.rsf.oracore_rsf is not present. An example parent directory used throughout this book is /u01/app/oracle/product/10.2.0/ em10g. Under this parent directory, the GC installer creates a separate Oracle home for each core component installed under the following subdirectories (created only if that component is installed):31 Grid Control Component Subdirectory Created below Parent Directory OMR db10g OMS oms10g OMA agent10g The Grid Control parent directory, if pre created, should be empty, and once installed, should only contain the GC Oracle home subdirectories listed above (post-installation; do not create additional subdirectories under this directory). If you select the GC New Database Option (which installs a new database for the Management Repository), the installer lays down all three Oracle homes under the parent directory. If you use the GC Existing Database Option (where a Management Repository is installed in an existing database), the installer creates two subdirectories under the parent directory: agent10g and oms10g. Both the GC New and Existing Database Options hard-code these subdirectories. When you have a choice of home directories, as when installing standalone Agents on target hosts, for consistency use the same OMA Oracle home as the hard-coded one for the chain-installed Agent (all the way down to the agent10g subdirectory)32 across all target nodes. As with the Grid Control installation of the OMS and chain-installed Agent, most standalone Agent installation methods create the subdirectory agent10g under the Parent Directory specified. The same applies if you choose the same Oracle home path for the OMR database software (even if it’s on a different host than the OMS) when using the Existing Database option as is hard-coded when using the New Database option. This ensures consistency for all GC Oracle homes across all hosts containing GC components. Remember not to use symbolic links in any directory specifications—Grid Control cannot parse them. 31 The agent10g subdirectory is also used when installing standalone Agents on target hosts using the GC Additional OMA Option. 32 See Chapters 5, 6, and 7 for more information on installing standalone Agents. 70 Oracle Enterprise Manager 10g Grid Control Implementation Guide Stop Database Listeners Using Port 1521 OMR If you choose the GC New Database Option, this creates a database and starts a default Oracle Net listener on port 1521 and the IPC key value EXTPROC. However, if you’re sharing the OMR database host with another Oracle product running a listener, and if it uses the same port or key value, the installer can only configure the new listener, not start it. Before kicking off the installer, shut down any existing database listeners on port 1521 to guarantee that the new database listener for the Repository database starts during the installation. You can determine whether an existing listener process is running, the listener name, and its Oracle home directory by executing the following command as the oracle user: ps -ef | grep tnslsnr … /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER -inherit If output similar to the following is returned, the listener is running. Here, /u01/app/oracle/ product/10.2.0/db_1 is the Oracle home directory where the listener is installed33 and the listener name is the default, LISTENER. Then, get the TCP/IP port and IPC key value from the listener.ora file or from a listener status command. If the listener uses port 1521, change it to another port number, or if needed, change the OMR port number after installation.34 Synchronize OS Timestamps/Time Zones OMR, OMS, OMA Ensure that the timestamp on the OMS host is synchronized with those on all other GC hosts, including the Repository and Agent hosts. Synchronized timestamps are particularly important for the following: ■■ When viewing GC log files in the Console or at the OS level ■■ Between RAC Repository nodes to prevent interconnect problems ■■ On Cluster Agent nodes to avoid Agent installation errors such as “/bin/tar: ./Apache/ Apache/conf/dms.conf: time stamp 2006-03-21 23:27:52 is 21 s in the future” You can make minor changes in time on RAC nodes, in the seconds range. To make larger time changes, shut down the instances on all RAC nodes. This avoids false evictions, especially if you’ve applied any 10g low-brownout patches for Bug 4896338, which allow for very low miscount settings. DBMS_SCHEDULER jobs will be affected by time changes, as they use actual clock time rather than System Change Numbers (SCNs). Apart from these issues, the Oracle server is immune to time changes (e.g., transaction and read consistency operations will not be affected).35 33 This may not be the Oracle home if TNS_ADMIN is set; in this case, get the Oracle home from the oratab file. This requires changing the port in the OMR listener.ora file and in the $OMS_HOME/sysman/config/emoms .properties file on the OMS for the properties oracle.sysman.eml.mntr.emdRepPort and oracle.sysman.eml.mntr .emdRepConnectDescriptor. 35 Note 200346.1 (Oracle internal only). 34 Chapter 2: Grid Control Preinstallation 71 The recommended approach to synchronize timestamps is to run the Network Time Protocol (NTP) on all GC hosts.36 All Grid Control and target hosts should use the same NTP server. (Even if you’re running the NTP daemon, spot-check that timestamps are synchronized across GC hosts by manually running the date command as simultaneously as possible on each host.) To check that the NTP daemon is running, execute the following command: /sbin/service ntpd status You can also check for the presence of the ntpd process, which will return output similar to that shown here: ps -ef | grep ntpd | grep -v grep ntp 1420 1 0 Sep29 ? 00:00:03 ntpd -U ntp -p /var/run/ntpd.pid If the NTP daemon is not running, start it as root: /sbin/service ntpd start For Linux hosts, you can add the “-x” flag to this ntp daemon startup command to prevent the clock from going backwards. Caution If you plan to use a RAC Repository database, you must run the NTP or an equivalent service to keep the clocks synchronized across RAC nodes. Otherwise, you may experience unexpected OMS shutdowns, and data collection timestamps may be incorrect (Bug 4500241). In addition to timestamps, set the OS time zone (and update the hardware clock) as desired for Grid Control on the OMS, OMR, OMA, and target servers. It is particularly important to set the OS time zone correctly on the node(s) to contain the OMR database, as the database time zone defaults to the OS time zone. If the OS time zone is not a valid Oracle Database time zone, the latter defaults to UTC.37 For instructions on changing the OS time zone, consult your platformspecific documentation. Tip If your GC implementation spans multiple time zones and is centrally managed, consider using the UTC time zone for all Grid Control hosts, including managed target hosts. This will help you avoid having to convert time zones when examining log files, and the like. 36 Each machine has a different clock frequency and, consequently, a slightly different time drift. NTP computes this time drift within approximately 15 minutes and stores this information in a drift file. NTP adjusts the system clock based both on this known drift and on a chosen time-server for all RAC nodes. 37 OracleMetaLink Note 330737.1 provides a list of the time zones that the 10g Oracle Database supports; this list is the same one the GC Agent supports. 72 Oracle Enterprise Manager 10g Grid Control Implementation Guide As an example, on Linux you can change the system time zone in a few short steps. Logged in as root, first check which time zone your machine is currently using by executing the following command, whose output will be in the format shown here: /sbin/clock Mon 11 Dec 2006 05:58:34 AM EST -0.097270 seconds In this case, EST is the current time zone. Say you wanted to change the time zone to UTC. Back up the previous time zone configuration, which for Linux is stored in /etc/localtime. (Don’t try to read this file, it contains binary data.) cp /etc/localtime /etc/localtime.bk Next, change to the directory /usr/share/zoneinfo. Here you will find a listing of time zone regions, both files and directories. If appropriate, change to the directory named after the appropriate region. If you live in the United States and want to use local time, you would change to the directory US or America (the latter is also valid for Canada and the other Americas) and choose your local time zone. In this example, you would not need to change directories because UTC is a file under /usr/share/zoneinfo. Copy the appropriate time zone to /etc/localtime, as shown in this example: cd /usr/share/zoneinfo cp UTC /etc/localtime If you have the utility rdate, update the current system time with this new time zone by executing the following: /usr/bin/rdate -s time.nist.gov Finally, set the hardware clock by executing: /sbin/hwclock --systohc Confirm Platform-Specific Software Requirements ✔ OMR, OMS, OMA OMR, OMS, and standalone OMA hosts have platform-specific requirements over and above the generic software requirements already mentioned. These requirements and the commands to manually check them are listed in the section “Platform-Specific Software Requirements” in Appendix B (online).38 Use this section of the appendix now to confirm and redress any outstanding prerequisites for your platform. This section begins by pointing you to the Release Notes to learn 38 Package and kernel requirements in the online Appendix B of this book are interpreted from the Oracle manual, Enterprise Manager Grid Control Installation and Basic Configuration, Appendix B: Platform-Specific Package and Kernel Requirements, cross-referenced with the platform-specific Oracle Enterprise Manager Grid Control Quick Installation Guides. The generic installation guide does not make it clear whether the requirements apply to the OMS, OMR, standalone OMA, or all of them; the platform-specific guides are clearer but still hazy on this point. Appendix B of this book clarifies the requirements for each GC component. Chapter 2: Grid Control Preinstallation 73 Don’t Forget to Run the Standalone Prereqchecker Reminder: If you haven’t yet run the Prereqchecker in standalone mode, run it now [see “The Prereqchecker” section in Appendix B (online) for instructions] to double-check that you’ve satisfied at least the subset of requirements that the Prereqchecker verifies. about platform-specific installation issues. The body of the section offers a breakdown by platform of required OS packages, patches, kernel parameters, and group and user rights. Summary This chapter is long because preparation is the key to installing Grid Control successfully and efficiently. You begin by downloading and staging the GC software. At the same time, you make some basic architectural design decisions, then address prerequisites related to network configuration, hardware, and software (both generic and platform-specific). Now you’re ready to move on to Chapter 3, where you actually install Grid Control. Prepare to reap the benefits of this thorough preinstallation groundwork. This page intentionally left blank Chapter 3 Grid Control Installation 75 76 Oracle Enterprise Manager 10g Grid Control Implementation Guide T his chapter gives you directions from start to finish on how to install the Grid Control Repository, OMS, and chain-installed1 Agent, using the first three installation types, which you may recall from Chapter 2 are as follows: Installation Type Nickname Enterprise Manager 10g Grid Control Using a New Database Installation Type GC New Database Option Enterprise Manager 10g Grid Control Using an Existing Database Installation Type GC Existing Database Option Additional Management Service Installation Type GC Additional OMS Option Chapters 5, 6, and 7 cover the fourth installation type, the GC Additional OMA Option, as well as the other methods for installing standalone Agents. To cover these three installation types, I’ve organized the chapter as follows: ■■ Detail the information you must gather as input to the installer ■■ Provide direction on how to consult platform-specific release notes to address installation bugs ■■ Describe how to initialize the OS environment for running the GC installer ■■ Explain three ways of running the GC installer—interactively, silently, or via the static ports feature—and how to monitor the installation ■■ Supply detailed instructions and screen shots for the three installation types when you run the GC installer interactively: ■■ For the GC New Database Option, I show all screen shots the installer presents. Roughly 75 percent of these screen shots also appear when you select the GC Existing Database and GC Additional OMS Options. ■■ For the GC Existing Database Option, I explain how to create, configure, and tune a new database to become the “existing” database, and show two screen shots unique to this installation type (rather than presenting duplicate screen shots to those for the GC New Database Option). ■■ For the GC Additional OMS Option, no screen shots are provided since they are a subset of those for the GC Existing Database Option. I explain what information the installer does not request compared with the GC Existing Database Option. ■■ Direct how to log in to the GC Console as a preliminary check to ensure the GC installation is successful (before installing any additional OMS hosts). Let’s get down to the first order of business. I strongly advocate that you determine your input to all installation questions before launching the GC installer. Such a fact-finding session accelerates the 1 See the introduction of Chapter 2 for an explanation of this term. Chapter 3: Grid Control Installation 77 installation process and prevents you from having to reinstall Grid Control because of insufficient preparation. Consultants and/or in-house IT staff who hold a single meeting to gather all required installer input exhibit more professionalism than those who start the installation blindly, only to have to repeatedly return to others for input. Gather Needed Installation Information OMR, OMS The installation information requested when you’re choosing any of the first three installation types differs by installation type as detailed in Table 3-1. Following are descriptions of the columns to help you navigate through the table. Starting with the top row, consult the columns in the order in which they are presented in the table: ■■ Columns 1 to 6 Find the column for your chosen installation type (columns 1, 2, 3) to determine whether it will display the Installation Screen Title (column 4) requesting the Input Field (column 5) described here (column 6). ■■ Column 7: Recommended Input Where applicable, I recommend an input value or choice based upon Grid Control best practices and Optimal Flexible Architecture (OFA)2 standards. ■■ Column 8: Is Input Mandatory or Optional? This column shows whether the requested input is mandatory or optional. I recommend entering all optional information now, but you can also configure it through the GC Console after the installation completes. ■■ Column 9: Figure Number These figures are the installation screen shots themselves found in the later sections for the GC New and Existing Database Options. These sections provide in-depth coverage of each input field. You don’t need to reference these figures now unless you need more detail than that provided in the table. After documenting the information you’ll enter when running the Grid Control installer for your specific installation type, you can move on to evaluating installation snags you may encounter, which depend upon the GC platform used, as described next. Address Installation Bugs OMR, OMS, OMA This book is intended to serve as a major source of information to navigate the Grid Control installation and configuration process. To make this experience as painless as possible, I try to minimize the need to bounce between this book and the Oracle GC documentation. You cannot ignore the Oracle GC documentation altogether. Release notes, which contain information on installation bugs, documentation errata, and the like, accompany each new software release for a specific OS, leaving us with a wide array of current release notes that would be impossible to incorporate into this book. To complicate matters, given that all platforms require the Full Installer for releases 10.2.0.1 through 10.2.0.3, then Patch Installers for 10.2.0.3 (the latest release available on some platforms) or 10.2.0.4, you must consult the respective release notes for both the Full Installer and Patch Installer needed at your site. 2 For more information on OFA standards, see Appendix D of the Oracle Database Installation Guide 10g Release 2 (10.2) for your Repository platform. 78 Applies to GC Existing Database Option Applies to GC Additional OMS Option Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes ProductSpecific Prerequisite Checks Yes No No Specify Configuration Yes No No Input Field Field Description Recommended Input Is Input Mandatory Figure or Optional? Number Specify Installation Type Choose GC New Database, Existing Database, or Additional OMS Option Choose desired installation type Site specific Mandatory 3-1 Specify Installation Location Parent Directory Creates Oracle homes db10g (only for GC New Database Option), oms10g and agent10g subdirectories under this base directory /app/ oracle/product/10.2.0/ em10g Mandatory 3-2 Product Languages Language Selection defaults to English Site specific Mandatory 3-3 Full path of Inventory directory (UNIX only) Screen appears only if this is the first Oracle installation on this host3 / oraInventory where = /app/oracle Mandatory 3-4 OS Group name (UNIX only) OS group must have write permission to Inventory directory oinstall or dba Mandatory Installation Screen Title Specify Inventory directory and credentials Mandatory 3-5 3-6 Repository Database Name New database name. Recommend using fully qualified database name. emrep. Mandatory Repository Database File Location Installer appends database name to the File location specified /oradata/ or if using ASM, +DATA/ /datafile/ Mandatory Oracle Enterprise Manager 10g Grid Control Implementation Guide Applies to GC New Database Option No No No Yes Yes No Yes No Yes Yes No Yes Yes No Specify Optional Configuration (UNIX only) Installer user must be a member of both groups to grant SYSDBA and SYSOPER permissions for creating the new database. dba dba Optional Database Hostname Port Service/SID SYS Password Use fully qualified hostname Listener port For single-instance use DB_NAME; for RAC use one INSTANCE_NAME (both nonqualified) Enter SYS password. . site-specific (default 1521) for single-instance, for RAC Site specific Mandatory Management Tablespace Location Configuration Data Tablespace Location Specify full path for datafiles. Cannot use Oracle Managed Files (OMF). /mgmt.dbf /mgmt_ ecm_depot1.dbf4 Mandatory Configure e-mail Notification? e-mail Address SMTP Server Check the box to specify e-mail address5 and SMTP Server name for SYSMAN alert notifications. Check the box Optional Configure MetaLink? Username, Password Check the box to enter MetaLink credentials to enable GC to search for and download patches.6 Check the box Optional 3-14 3-7 Grid Control Installation Information Needed Based on Installation Type (Continued) 3456 3 79 On Windows, the Inventory location defaults to :\Program Files\Oracle\Inventory\. If you’re using ASM, is +DATA/ /datafile. 5 You will likely receive multiple notifications following installation, triggered from out-of-box alerts. So initially, enter the e-mail address of the GC administrator responsible for tuning alerts. 6 I recommend creating a dedicated MetaLink account not tied to a person so that it remains open indefinitely and so that the account password is not inadvertently changed. 4 Grid Control Installation table 3-1. Specify Repository Database Configuration OSDBA group OSOPER group Chapter 3: Yes 80 Applies to GC Existing Database Option Applies to GC Additional OMS Option Yes Yes Yes Yes Yes (under Specify Passwords screen) Yes (under Specify Repository Database Configuration screen) Yes No No Installation Screen Title Specify Security Options Input Field Field Description Recommended Input Is Input Mandatory Figure or Optional? Number Configure Proxy? Proxy Server, Port Do Not Proxy For Proxy Username, Password, Realm Check the box if OMS must go through a firewall to connect to Agents or to MetaLink. Specify fully qualified hostname and port for proxy server. Enter internal address(es) not to proxy for, commaseparated such as .us .mycorp.com,.mycorp.com. Enter username, password, and realm for proxy server if it requires authentication. Site specific Optional Agent Registration Password Require secure communication for all Agents? Specify registration password for Agents to securely communicate with OMS. Check the box to require that the OMS communicate only with secure Agents. Enter a password Check the box Mandatory Use same or different passwords for accounts? SYS, SYSTEM, SYSMAN, DBSNMP Use same or different passwords as dictated by security policies. Password(s) must be fivecharacters minimum plus one number and start with a letter. Site specific Mandatory 3-8 Oracle Enterprise Manager 10g Grid Control Implementation Guide Applies to GC New Database Option Yes Oracle Configuration Manager Registration Enable OCM, CSI, MetaLink Account Username, Country Code, Connection Settings, Test Registration Register for OCM by supplying all fields to provide Global Customer Service with system configuration information. Check Enable Oracle Configuration Manager and complete remaining fields Optional No Yes Yes (under Specify Repository Database Configuration screen) Specify Passwords Agent Registration Password Require secure communication for all Agents? Specify a registration password for Agents to securely communicate with OMS. Check the box to require that the OMS communicate only with secure Agents. Enter password Check the box Mandatory No Yes Yes (under Specify Repository Database Configuration screen) SYSMAN Password Enter new password for SYSMAN (user should not exist). Site specific Mandatory Yes Yes Yes Summary Click Install Summarizes installation based on all input Click Install if all options are correct Mandatory Yes Yes Yes Install (Execute Configuration scripts) root password Need root password to run scripts orainstRoot .sh and allroot.sh. Site specific Mandatory table 3-1. Grid Control Installation Information Needed Based on Installation Type (Continued) 3-9 Grid Control Installation Yes Chapter 3: Yes 81 82 Oracle Enterprise Manager 10g Grid Control Implementation Guide Release notes for the base 10.2.0.1 release on OTN are available in the Oracle Enterprise Manager Documentation Library at http://www.oracle.com/technology/documentation/oem.html. Release notes for the latest Patch Installers are bundled with the platform-specific GC software that you already staged (see Chapter 2, “Stage the Grid Control Software”). These release notes are also put out as a separate OracleMetaLink consolidated document called Oracle Enterprise Manager Grid Control Release Notes for 10g Release x (10.2.0.x.0), where x is the latest release family. (This consolidated document also contains the Grid Control List of Bugs Fixed for that platform.) Before installing Grid Control, inspect the Installation and Upgrade Issues under the Known Issues section of both the Full and Patch Installer release notes for bugs and workarounds. To give you an idea of the types of installation bugs you may run across, following are some Linux bugs I’ve encountered (and workarounds) in various GC 10.2.0.x releases.7 (Most of my Grid Control installation testing has been on the Linux platform.) Some of these bugs are documented in the GC 10.2.0.x release notes for Linux, and others are referenced in OracleMetaLink notes cited in the footnotes for this chapter. ■■ The Grid Control OUI prompts you to install Oracle Configuration Manager (OCM), if desired. On UNIX, if the oracle user isn’t authorized to use CRON, OCM software is installed but OCM setup fails, as revealed only in the OUI installation log files. The reason is that the OCM setup process needs to add an entry to the oracle crontab to run data collections every 15 minutes. Do one of the following, depending on whether you can authorize oracle to use CRON: ■■ Enable oracle to use CRON by adding the oracle user to /var/adm/cron/cron.allow file (which requires root permission). Be sure to unset CCR_DISABLE_CRON_ENTRY before running the OUI as it is also documented that setting this variable to any value enables it. ■■ As a workaround to not using CRON, set the environment variable CCR_DISABLE_ CRON_ENTRY to 1 before launching the Grid Control OUI. Schedule OCM to run using a scheduling method other than CRON or run OCM manually whenever desired, as follows (ORACLE_HOME is the home in which OCM is running):8 $ORACLE_HOME/ccr/bin/emCCR $ORACLE_HOME/ccr/bin/emCCR $ORACLE_HOME/ccr/bin/emCCR $ORACLE_HOME/ccr/bin/emCCR $ORACLE_HOME/ccr/bin/emCCR start collect upload status stop ■■ Before installing Grid Control on Linux x86, make sure to unset all NLS-related OS environment variables (NLS_LANG, ORA_NLS, ORA_NLS33, ORA_NLS10, etc). Setting NLS environment variables can cause the Grid Control installation to fail.9 7 Some of these bugs may have been fixed since the publication of this book. To confirm this, consult the latest GC release notes for Linux. 8 See Note 369619.1. 9 See OracleMetaLink Note 272493.1. This was confirmed for GC 10.1.0.2 on Linux x86, but later releases on other platforms may be affected as well. Chapter 3: Grid Control Installation 83 ■■ The following bugs are specific to SUSE Linux Server Enterprise 9 (SLES 9) as the installation platform for the OMS: ■■ /usr/lib/libdb.so.2 is not available on SLES 9, which causes problems for the Grid Control Oracle Universal Installer (OUI). The solution is to create the following symbolic link before launching the OUI: ln -sn /usr/lib/libdb.so.3 /usr/lib/libdb.so.2 ■■ If you plan to select the GC New Database Option on SLES 9, first manually create an empty /etc/oratab file. The installer reports an error if this file is missing and does not create a new oratab file. ■■ During the execution of root.sh, the installer tries to create a hard link from /etc/ rc3.d/S99gcstartup to the file /etc/init.d/gcstartup. However, on SLES 9 the /etc/ rc3.d directory does not exist. The equivalent directory is /etc/init.d/rc3.d. Therefore on SLES, create the following hard link manually after the Grid Control installation completes: ln /etc/init.d/rc3.d/S99gcstartup /etc/init.d/rc3.d/gcstartup ■■ No Grid Control components restart automatically on reboot. You must manually start all EM components, including the OMR database and listener if created using the GC New Database Option, the OMS, and chain-installed OMA. A workaround is to implement the orarun script available from SuSE.10 Initialize the Oracle User Environment OMR, OMS, OMA Before running the Grid Control installer, you need to initialize the environment from which you will launch the OUI as the oracle user (or other user chosen to own the GC software). This initialization involves setting up X Server for UNIX platforms, then setting or unsetting the appropriate environment variables. Set Up X Server (UNIX Only) To run the Grid Control Oracle Universal Installer (OUI) interactively on UNIX, you need to set up X Server on the installer host. If you are on a UNIX platform, you can either use a PC with X Server software installed or a UNIX workstation. Note You can also run the installer silently, in which case you don’t need to install X Server libraries or run X Server. See the section “Silent Installation Method” later in the chapter for more information. If you’re using a PC with X Server software, start the X Server on the PC. If using a UNIX workstation, start a terminal session such as an X terminal (xterm) to the remote UNIX system where you want to install the software. Either way, log in directly as the oracle user—if you use su or sudo to log in as oracle, it’s likely the X Server software won’t work. 10 See OracleMetaLink Note 266049.1. 84 Oracle Enterprise Manager 10g Grid Control Implementation Guide UNIX Shell Variations Initializing the environment on UNIX platforms varies according to the UNIX shell you use. There are four main UNIX shells in use today. The following list identifies the shell startup script name, shows how to run the startup script, and provides the command syntax for setting environment variables for each shell. (Note that I use the Bash shell in examples throughout this book.) Bash Shell (bash) Bourne Shell (sh) Korn Shell C shell (csh (ksh) or tcsh) User’s Startup Script for Shell .bash_profile .profile .profile .login How to Execute Startup Script from User’s Home Directory . ./.bash_profile . ./.profile . ./.profile source ./.login Syntax to Set x=1 (Set Environment Variable x to “1”) x=1; export x x=1; export x x=1; export x setenv x 1 note Leave the session open where you’re logged in directly as the Oracle user, as you will be initializing the environment and launching the Grid Control installer in this session. Allow the remote UNIX host to display X applications on the local X Server by executing the following command on the remote host (use the fully qualified host name of the remote host): xhost To set the environment as indicated below, first determine the default shell for the oracle user: echo $SHELL Open the shell startup file for the oracle user in a text editor, such as vi. Refer back to the list in the “UNIX Shell Variations” sidebar above for the shell startup file name. The example here assumes a Bash shell startup file is used. vi .bash_profile Enter or edit the following line to set the default file mode creation mask to 022 in the shell startup script. This ensures that the Oracle installation lays down files with 755 permissions: umask 022 Save the shell startup file and exit the editor. Then run the shell startup script (again, we use the Bash shell in examples throughout this book): . ./.bash_profile Chapter 3: Grid Control Installation 85 Enter the following command to direct X applications like the OUI to display on your local PC:11 DISPLAY= :0.0; export DISPLAY where is the host name or IP address of your local PC. Test Xwindows functionality by verifying that when you run the xclock command on the UNIX host, a clock appears on your X Server display. The location of xclock varies by UNIX platform as follows: Platform xclock Command AIX /usr/bin/X11/xclock HP-UX /usr/bin/X11/xclock Linux /usr/X11R6/bin/xclock Solaris /usr/openwin/bin/xclock Set/Unset OS Environment Variables The following environment variables should be set or unset for GC installation as noted here. 12 Variable Name Value Description CCR_DISABLE_ CRON_ENTRY 1 Set only if you want the GC OUI to install Oracle Configuration Manager and the oracle user isn’t authorized to use CRON. LANG, NLS_ LANG, and other NLS variables Unset These variables might be incompatible with the OS default locale and OMR database character set.12 OPATCH_NO_ FUSER Unset (for GC 10.2.0.2 or higher) Opatch (called by OUI) uses the fuser executable to detect active processes. Unset if fuser is present in / sbin or /usr/sbin or is in the PATH. FALSE (for GC 10.2.0.1 or lower) This is a workaround for Bug 4898745 (fixed in GC 10.2.0.2) to bypass the OPatch active process checks. Set Set to /u01/app/oracle, for example. This is not required, but the OUI picks up this location to pre populate the EM Parent and oraInventory directories. ORACLE_BASE 11 You do not need to set the DISPLAY environment variable for GC silent installations. This is a recommendation. However, the installation will fail if you don’t unset NLS variables on Linux for GC 10.1.0.2, and perhaps for other releases and platforms. See OracleMetaLink Note 272493.1. 12 86 Oracle Enterprise Manager 10g Grid Control Implementation Guide Variable Name Value Description ORACLE_HOME Unset The OUI prompts for a parent directory, which contains multiple Oracle homes.13 ORACLE_SID Unset OUI will prompt you to enter the OMR database name. PATH Do not include Oracle paths. Include fuser executable in PATH if it’s not in / sbin or /usr/sbin. This variable should not contain any paths for installed Oracle products, but only standard OS paths for all users. TMP and TMPDIR (UNIX), TEMP, TMP (UNIX and Windows) Unset (set if required) OUI uses this directory for executables and link files. Unset if free space in temporary directory is at least 250MB and won’t be purged. Otherwise set to an alternate directory that meets these specifications. TZ Unset Set desired time zone at OS level instead. As this table shows, you should set ORACLE_BASE (not required but recommended), and set OPATCH_NO_FUSER and temporary directory variables only as needed. Unset all other Oraclerelated variables not specifically mentioned. 13 On UNIX, consult the list in the “UNIX Shell Variations” sidebar above for the syntax to set or unset variables in the various UNIX shells. To confirm that the UNIX environment has been set correctly, enter the following commands as the oracle user: umask env | sort Verify that the umask command displays a value of 22, 022, or 0022 and that the environment variables you set in this section have the correct values. On Windows, open a command window and use the following syntax to set or unset variables, respectively: set = set = Leave the command shell open to execute the GC installer (setup.exe) later, thereby allowing the OUI to pick up all environment variable settings. To verify whether the Windows environment is set correctly, execute the following command from the command shell: set 13 Multiple Oracle homes are installed under this parent directory for the OMR, OMS, and OMA, depending on the installation type chosen. Chapter 3: Grid Control Installation 87 Running the Installer with Desired Options Before kicking off the Grid Control installation, it’s always a good idea to consult the Readme instructions for your platform that accompany the Full Installer version you staged in Chapter 2. The README.txt is located in the top level directory of the distribution. Scan it and you’ll see that for a fresh installation of Grid Control, it says to “Run runInstaller (or setup.exe for Windows) to start installation.” Let me improve the Readme by adding, “with the appropriate parameters.” Two options that require parameters are the silent installation method and the static ports feature. This section is laid out as follows: ■■ Silent Installation method ■■ Static ports feature ■■ Starting and monitoring the installation Both options have limited usefulness, so I explain them only briefly. I also explain how to start and monitor your installation, which is the same regardless of the installation type you select. Note If you are planning to install the OMS or OMR in an Active/Passive Cold Failover Cluster (CFC) environment, see the following sections in Chapter 15 for alternate Grid Control installation instructions: “Install OMSs in an Active/Passive Configuration” and “Install the Repository in a Cold Failover Cluster.” If a server load balancer is available now for use with Grid Control, also see “Use a Server Load Balancer” in Chapter 15, as it requires a different approach to GC installation. When reading about the silent installer and the static ports feature, for sake of completeness, I provide the syntax for their respective parameters. However, don’t kick off the installer until you get to the appropriate section for your installation type. Silent Installation Method If you have good reason not to run the Grid Control installer interactively, run it silently. Silent installations work for all four installation types. What reason might you have for using the silent installer? ■■ If you can’t run X Server on the host where running the installer. The X Server libraries may not be installed, or your corporate security policy may not allow you to run X Server on the host in question. ■■ If you are installing many GC environments. The silent installer eliminates the need to interact with the installation. This may reduce installation time appreciably when building multiple Grid Control environments, such as at each corporate satellite office or in a classroom setting. (In my opinion, the silent installer adds unnecessary complexity when laying down just one or two GC environments.) ■■ If you are installing many standalone Agents on target hosts. (The fourth installation type, the GC Additional OMA Option, is not covered in this chapter. I detail Agent deployment using the silent installer in Chapter 7, along with the other methods of installing a standalone Agent.) 88 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ If you want to better guarantee the reproducibility of the installation, such as to ensure identical production and test environments, use the silent installer in conjunction with the static ports feature. This is particularly useful if multiple administrators are running the installer. To perform a silent installation, start by editing a response file template for your chosen installation type, supplying the same information that the OUI screens request. The response file takes the place of entering this information interactively. The four response file templates, one for each installation type, are located in the Disk1/response directory below the top level directory of the GC distribution. To run the Grid Control installer silently, you would simply launch the GC installer executable used for interactive installation with the following parameters, one of which specifies the edited response file (remember, don’t actually run the installer yet): On UNIX ./runInstaller-responseFile \ -silent -waitForCompletion On Windows .\setup.exe -responseFile -silent For complete instructions on running Grid Control silent installations of the first three installation types (that deploy an OMS), see Chapter 4 of Oracle Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 2 (10.2). Static Ports Feature I can think of two principal reasons for considering the static ports feature. ■■ If you’re installing an additional OMS. Using static ports is the only way to make the port configuration identical across OMS servers. For some reason, if you don’t use static ports, Grid Control only assigns some—not all—default ports for additional OMSs, even if these ports are available. ■■ If one or more default OMS ports are not free for any OMS. However, this should not be the case if you dedicate the OMS host to Grid Control. You can use the static ports feature with either the interactive or silent installer. The silent ports feature allows you to specify custom port numbers for the Console, OMS, and chaininstalled Agent components to override default port numbers that the installer otherwise assigns.14 Post installation, you can reconfigure any GC component to use a different port, but the static ports feature makes it easier to use custom ports up-front. The static ports feature relies upon your editing a template file called staticports.ini located in the Disk1/response directory below the top level directory of the GC distribution. (This directory is also where the silent installation response file templates are found.) Example custom port entries in the file are as follows: Enterprise Manager Central Console Port=5345 Web Cache HTTP Listen port=7799 14 See OracleMetaLink Note 353736.1 for a listing of the default EM 10.1 and 10.2 ports. Chapter 3: Grid Control Installation 89 When specifying the static ports option to the installer, it uses the values from the file instead of the default port numbers. The EM documentation states that default port numbers are used for any components not explicitly listed in the staticports.ini file. However, my own experience with this feature suggests otherwise. I highly recommend that you specify all components explicitly in this file, assigning each component a default or custom port number, whichever the case may be. The staticports.ini file uses the same general format as the file $OMS_HOME/install/portlist.ini, which the GC installation creates. Therefore, if you install Grid Control, with either default or custom ports, and you wish to use the same port numbers to install an additional OMS, use the ports listed in the portlist.ini file from the first installation in the staticports.ini file for additional OMS installations. If copying and pasting entries from portlist.ini, comment out the header information with a hash mark (“#”) or remove it altogether. When you finish editing the staticports.ini file, you are ready to install Grid Control with the static ports feature. When the time comes to run the installer, either interactively or silently, enter the following at the command line: ./ -staticPortsIniFile /staticports.ini For more details on the static ports feature, see Chapter 4 of Oracle Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 2 (10.2). Starting and Monitoring the Installation You are ready now to learn how to start and monitor your installation. Then you can go to one of the following sections, depending on the installation type you want to choose, and actually kick off the installer: ■■ Install Grid Control Using a New Database ■■ Install Grid Control Using an Existing Database After choosing one of these installation types, continue with the remaining sections: ■■ Log in to GC Web Console ■■ Install an Additional OMS (if applicable) When launching the Grid Control installer, you will run it interactively or silently using either default or static ports. Most of you will be installing Grid Control interactively rather than with the silent installer. You will probably not need the static ports feature unless you’re installing an additional OMS. Whatever installation type you choose, remember to start the installer in the same command shell you opened and initialized in the earlier section “Set/Unset OS Environment Variables.” You will execute one of the following commands, depending on the platform (with the appropriate parameters for a silent installation and/or the static ports feature): On UNIX cd ./ /runInstaller On Windows cd .\ \setup.exe 90 Oracle Enterprise Manager 10g Grid Control Implementation Guide The varies based on the location of the installation software: ■■ If running the OUI from DVD, = /Disk1. When starting the installer from a command line, you should not be in or below the DVD mount point directory, as the installer process will tie up that directory, prohibiting you from ejecting the DVD when prompted to insert Disk 2. ■■ If installing from a staged location on disk, = /Disk1. You can also dot execute the installer from within its own directory, i.e., “./runInstaller” or “.\setup.exe”. Regardless of the installation method you opt for—interactive or silent—at some point the installer will take over and the process behind the scenes will become identical. That point for interactive installations is when you click “Install” after completing the interview process in the OUI screens (with or without using the static ports feature), or if running the installer silently, after launching it. The installation time for the base GC release will vary according to your architectural choices, the installation type you select, the existing database in which you’re installing the OMR, and the available hardware resources. Following is a rough estimate of the total running time for each installation type: Installation Type Installation Running Time (Approximate) GC New Database Option 2 hours GC Existing Database Option 1.5 hours GC Additional OMS 1 hour Don’t be too hasty to terminate an installation that may appear to be hung because of the limited output from the Java installer. Don’t suffer in silence; you can find out what’s happening behind the scenes by tailing the installation log files using the command tail -f on UNIX systems (for Windows a UNIX style tail command is also available in the Windows 2003 Resource Kit). Here are some notable log files: ■■ /logs/installActionsYYYY-MM-DD_HH-MI-SS-[A-P]M.log This main installer log file is also echoed to stdout. It is useful to consult this log file to review previous installer activity.15 ■■ $OMS_HOME/sysman/install/log/emca_repos_createHH_MI_DD.log This is a useful log file for a particularly long operation summarized (with spelling error to boot) in the Java output as “Operation Creating OMS Respository [sic] is in progress.” 15 On UNIX platforms, the directory is specified by the Inventory pointer location file, oraInst .loc, if it exists. If it doesn’t exist, you specify when installing Grid Control on the screen “Specify Inventory directory and credentials.” On Windows platforms where no Oracle products have been installed, the inventory directory is located under \oraInventory\ if ORACLE_BASE is set in the environment where the installer was launched. If at least one Oracle product was previously installed (depending upon the product), can be :\Program Files\Oracle\Inventory\. Chapter 3: Grid Control Installation 91 ■■ $ORACLE_HOME/cfgtoollogs/cfgfw These are log files for each Configuration Assistant, located under the database Oracle home. ■■ cfmLogger_YYYY-MM-DD_HH-MI-SS-[A-P]M.log All configuration logs are appended to this log file. Note The installer is touted to have a resumability feature if it unexpectedly terminates. The installation should pick up where it left off, but don’t count on this feature working. In testing, I was not able to demonstrate this functionality. The installer appeared to start from scratch and when I specified the same parent directory used in the failed installation, it complained about the existence of the previous EM installation and asked me to remove it. Install Grid Control Using a New Database If you want to install Grid Control using the GC New Database Option, it’s finally time to kick off the installer. Go ahead, don’t be shy. I’ll wait. (See the earlier section “Starting and Monitoring the Installation.”) The GC New Database Option is the default installation type selected when you run the installer. This option lays down all Grid Control components on a single host. A new Oracle Database 10g (10.1.0.4) Enterprise Edition is created with a Repository, and the OMS and tag-along OMA are deployed. Following are the screen shots presented when you choose this installation type. Note Many of these screens below from the GC New Database Option also appear in the other two Grid Control installation types. Therefore, for reference purposes each screen is labeled “New,” “Existing,” and/or “Add’l” to indicate which installation types display the screen. Specify Installation Type As shown in Figure 3-1, select Enterprise Manager 10g Grid Control Using a New Database, and click Next. Specify Installation Location Use the screen shown in Figure 3-2 to specify a parent directory under which the installer will create three subdirectories, one for each Oracle home: agent10g, db10g, and oms10g. This directory cannot be a symbolic link. The base directory of the parent directory will default to ORACLE_BASE/ GridHomes if ORACLE_BASE is defined. Change the directory specification as desired. In this book, /u01/app/oracle/product/10.2.0/em10g is used as the parent directory. It is better to let the installer create the parent directory for you, as it will use the desired permissions. The products are installed in English by default. Click Product Languages to install in different language(s). Otherwise, click Next. The next screen shows the Product Languages screen. 92 Oracle Enterprise Manager 10g Grid Control Implementation Guide FIGURE 3-1. Specify Installation Type: New, Existing, Add’l FIGURE 3-2. Specify Installation Location: New, Existing, Add’l Chapter 3: Grid Control Installation 93 Note Screens such as the one above and throughout the book are taken from Grid Control installations on UNIX, so they contain UNIX style file paths (forward slashes). If installing Grid Control on Windows, use Windows-style drive specifications (such as C:) and file paths (backslashes). Language Selection This screen (see Figure 3-3) provides 14 languages from which to choose. You can choose multiple languages, not for the installation itself, but for subsequent use of the product. The installer installs text in the selected languages and the fonts required to display these languages. Select the language(s) desired and click OK. Then click Next on the Specify Installation Location screen. Administrators must add the desired languages in their browsers used for Console access to display the GC web pages in these languages. Taking Internet Explorer as an example, select Tools, Internet Options, click Languages…, click Add…, select the additional languages, and then click OK three times. Specify Inventory Directory and Credentials Enter the path to the inventory directory, as shown in Figure 3-4. This path does not need to exist, but the oracle user must have permissions to create it. In this illustration, the oinstall group is used as the Operating System group name. You can also use the dba group, depending on your site’s Oracle standards. This screen will not appear on Microsoft Windows platforms, where the inventory location defaults to :\Program Files\Oracle\Inventory. Click Next. FIGURE 3-3. Language Selection: New, Existing, Add’l 94 Oracle Enterprise Manager 10g Grid Control Implementation Guide FIGURE 3-4. Specify Inventory Directory and Credentials: New, Existing, Add’l If on the above screen you receive the error, “OUI-10182: The effective user ID does not match the owner of the file, or the process is not the super-user, the system indicates that superuser privilege is required,” it is probably due to insufficient privileges to create the oraInst. loc file in the platform-specific default location. To resolve, either pre create this file or grant the OS user running the installer (or the OS group you specify on this screen) write permissions to the directory where oraInst.loc is created. This should enable you to continue without having to restart the installation. Product-Specific Prerequisite Checks Figure 3-5 shows the prerequisite check, type (Automatic, Optional, or Manual), and status (Succeeded, Warning, Skipped, or Failed). ■■ Click Next if all prerequisite checks are successful, which should be the case if you ran the Prereqchecker in standalone mode without error as instructed in Chapter 2. ■■ Let the prerequisite checks finish, then click each failed prerequisite check to view its corresponding details at the bottom of the screen. These details show expected vs. actual results, error messages, and instructions to resolve them. Fix the failed checks manually if possible, and if fixed, select the check and click Retry. After resolving all checks as best you can, click Next. It is best to resolve all errors before continuing. However, you can click Yes to ignore the errors and attempt to proceed with the installation, although the installer may not allow you to continue, depending upon the severity of the error. Chapter 3: FIGURE 3-5. Grid Control Installation 95 Product-Specific Prerequisite Checks: New, Existing, Add’l Specify Configuration Enter the desired Repository database name, datafile location, and OS groups in the screen shown in Figure 3-6. Fully qualify the database name you choose. The Groups Specification section is grayed out for Windows platforms. Click Next. Specify Optional Configuration Figure 3-7 is optional in that you don’t need to complete it for the installation to succeed and all fields are disabled by default. However, you must specify these input fields if you want Grid Control to send you e-mail alerts or help you patch Oracle software. “Optional” refers to whether you want the installer to configure this functionality now or subsequently perform this configuration yourself through the Console. The intent of making this configuration optional is so you don’t have to delay a GC installation just because your SMTP server, OracleMetaLink credentials, or proxy server are not ready, or because an OMP or OMS server are not on the network yet. But if these external elements are ready (and they will be if you followed the preinstallation steps in Chapter 2), why not let the GC installer configure them rather than having to do this yourself after the installation? ■■ Configure E-mail Notification Checking the box and entering the required information here configures a global e-mail notification method, not only for the built-in SYSMAN account, but for any GC users you later create. SYSMAN is the owner of the Repository database schema and a super administrator (e.g., given full privileges) who is automatically configured to receive certain out-of-box alerts. 96 Oracle Enterprise Manager 10g Grid Control Implementation Guide FIGURE 3-6. Specify Configuration: New FIGURE 3-7. Specify Optional Configuration: New, Existing Chapter 3: Grid Control Installation 97 ■■ E-mail Address Enter the e-mail address to associate with the SYSMAN account. The installer does not verify that this e-mail address is valid, so you should confirm this yourself. ■■ SMTP Server The installer checks that the SMTP server is reachable, and doesn’t let you continue unless you enter a valid SMTP server name or uncheck the box, Configure E-mail Notification. ■■ Configure MetaLink Check this box to enter a valid OracleMetaLink username (an e-mail address) and password for Grid Control to use when searching for and downloading patches. The installer does not verify the validity of these credentials. Ideally, you should set up a dedicated account for Grid Control that is not tied to a particular person.16 This will prevent connectivity problems due to a user password change or to an account that is closed when someone leaves the company. After the installation, you can also enter these credentials through the Console by navigating to Patching Setup under Setup. ■■ Configure Proxy Check the box if the Management Service must go through a firewall to connect to OracleMetaLink or to an Agent on a target host. If you click Test Proxy, the installer confirms whether the proxy server is reachable. ■■ Proxy Server Specify the fully qualified hostname for the proxy server. ■■ Port Enter the port for the proxy server, typically port 80 or 8080. ■■ Do Not Proxy For Enter internal address(es) not to proxy for, separated by commas but no spaces, such as .us.mycorp.com, .mycorp.com. ■■ Proxy Username, Password, Realm If the proxy server requires authentication, enter a username, password, and realm for the proxy server. Click Next to advance to the next screen. Specify Security Options In Figure 3-8, you specify an agent Registration password to enable Agents to communicate securely with the OMS via HTTPS as well as HTTP. Also, check the box “Require Secure Communication for all agents” to lock the OMS, thereby insisting upon such secure communication. Not checking the box configures the OMS in an unlocked state, i.e., both secure and unsecured Agents can talk to the OMS. On the bottom half of the screen, you can choose to use different passwords for all database accounts or to use the same password, as your company’s security policy directs. The password for ias_admin (the administrative user for the Oracle Application Server Console) is set to that for SYSMAN. Both Grid Control administrator and database passwords must abide by the following restrictions (unless indicated otherwise for one or the other):17 ■■ Passwords must be between 5 and 30 characters long. ■■ Passwords must start with an alphabetic character—they cannot start with a number. ■■ GC administrator passwords must include at least one integer. 16 To create a MetaLink account, go to http://metalink.oracle.com, click Register for MetaLink under First-Time Users, and complete the interview process. You need a valid Customer Support Identifier (CSI) number to create an account. 17 This password scheme for GC administrators is admittedly the weakest in the database network, due to Grid Control Bug 4113146. 98 Oracle Enterprise Manager 10g Grid Control Implementation Guide FIGURE 3-8. Specify Security Options: New, Existing, Add’l ■■ GC administrator passwords can only contain alphanumeric characters and “_” (underscore). ■■ Database passwords cannot use Oracle reserved words,18 and I discourage their use for GC administrator passwords as well. Click Next. Tip You may be tempted to leave the OMS unlocked if all Grid Control components are behind the corporate firewall. Do not let this lull you into a false sense of security. Roughly 70 percent of security threats come from within (from hackers, internal theft, and the like). A fundamental security concept is that of multiple layers of protection. Avail yourself of this simple security layer and lock the OMS. The only cost is Agent Registration password management. 18 For a list of Oracle reserved words, see Table G-1 in Appendix G of Oracle Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 4 (10.2.0.4.0). Chapter 3: Grid Control Installation 99 Oracle Configuration Manager Registration As of GC 10.2.0.2, the OUI is bundled with Oracle Configuration Manager (OCM), which is installed in the OMS and chain-installed Agent homes, collects configuration information for these homes and the host, and uploads it regularly to Global Customer Support (GCS) (“Oracle Support”) to help improve its technical support service. Completing the screen shown in Figure 3-9 is more convenient than manually installing and configuring OCM later. (This is covered in Chapter 8 under the heading, “Install Oracle Configuration Manager Client.” However, as this section points out, you must use the Command Line Interface to install and configure the OCM in the OMR Database home.) ■■ Enable Oracle Configuration Manager If you check this box to enable OCM now, you must complete all fields on this screen. When you click Next, the OUI prompts you to click Accept License Agreement. ■■ Customer Identification Number (CSI) Enter your Customer Support Identifier (CSI) here. If registration is successful, the OracleMetaLink administrator of this CSI will receive an e-mail confirmation that OCM has been enabled on this server. ■■ Metalink Account Username Enter the username of the administrator associated with this CSI, which must match Oracle’s records. FIGURE 3-9. Oracle Configuration Manager Registration: New, Existing, Add’l 100 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ Country Code Enter the country code for the MetaLink Username. To confirm, log in to OracleMetaLink as this user and see the Profile section under the Licenses link for the Country Code associated with the CSI. ■■ Connection Settings Click this button if you must connect from this host through a proxy server to reach the Internet. You are prompted for the name and port of the proxy server, and if the proxy server requires authentication, the proxy username and password. ■■ Test Registration Click this button to test the connection between the host and the OCM service. You cannot proceed past this screen until the information has been verified, so you may as well test the registration using this button. If the test fails, and you cannot obtain the correct OCM information at this time, uncheck Enable Oracle Configuration Manager and configure OCM manually later on as instructed in Chapter 8. A log of all OCM installation steps and errors is located under the directory $OMS_HOME/ ccr/log. Click Next to continue. Summary Verify the options you chose in Figure 3-10. If everything is correct, click Install. FIGURE 3-10. Summary: New, Existing, Add’l Chapter 3: FIGURE 3-11. Grid Control Installation 101 Install and Execute Configuration Scripts: New, Existing, Add’l Install An installation screen (see Figure 3-11) shows a progress bar. Approximately 30 minutes into the installation, the Execute Configuration Scripts dialog box pops up for UNIX platforms (it does not appear on Windows platforms). If this is the first Oracle installation on the server, you are prompted to log in to the OMS host in a separate session as root and run orainstRoot.sh under the Inventory directory. The installer always prompts you to execute allroot.sh under the database home as root. This script in turn executes three root.sh scripts, one for each of the Oracle homes installed. Provide default answers to all questions these shell scripts ask by hitting return at each prompt. Return to the dialog box and click OK to continue. Configuration Assistants The screen in Figure 3-12 appears when the assistants begin running. The screen shows the name, status, and type (Recommended or Optional) of each configuration assistant. If an assistant fails, you can select that assistant, click Stop, fix the problem, then click Retry. Depending on the severity of the error, and the dependency of one assistant on another, the installer may not give 102 Oracle Enterprise Manager 10g Grid Control Implementation Guide FIGURE 3-12. Configuration assistants: New, Existing, Add’l you the chance to continue to the next assistant. In this case, you must either click Cancel, or manually kill the installer. Regardless of how many configuration assistants fail, if the installer gets through all of them, it will not report the installation as having failed at the end. This is misleading, as failure of any of the assistants leaves the installation in a very questionable state. You should seriously think about de installing Grid Control even if only one assistant fails. It is better to address the problem and re install cleanly, or GC may not function properly, either immediately or down the line after much post configuration effort. Once you input the information on the installer screens and click Install, it takes about half an hour for the installer to lay down the OMR, OMS, and OMA software trees. The configuration assistants then run for another 1.5 hours or so when you select the GC New Database Option. Most Configuration assistants run for less than 5 minutes, except that for the Oracle Database (which runs for about 15 minutes), and the OMS Configuration (which runs for about 30 minutes). Click Next if all configuration assistants succeed, or one or more fail and you are willing to take a big chance (in which case you should consider another line of work). Chapter 3: FIGURE 3-13. Grid Control Installation 103 End of Installation: New, Existing, Add’l Note You can also run a configuration assistant in standalone mode for some OUI errors, as instructed in the error message. However, it is typically easier and definitely more reliable to de install Grid Control entirely, resolve the problem that caused the failure, and rerun the installer from the beginning. End of Installation This screen (Figure 3-13) appears with the URL for how to access the Console, the Repository connection details, and the path to a file containing the Release Notes. Click Exit. A dialog box asks you to confirm if you really want to exit. Click Yes. Install Grid Control Using an Existing Database Before you can install Grid Control using an existing database, you must first install and configure a new database (or configure an existing database)19 to contain the Management Repository. 19 Instead of building a new database for the OMR, you could use a preexisting database, but this would likely occur only in an upgrade scenario when using an OMR database from an earlier GC release. GC upgrades are outside the scope of this book, so configuration of a preexisting database is not covered here. 104 Oracle Enterprise Manager 10g Grid Control Implementation Guide In this section we build a new database,20 which then becomes the “existing” database where the Repository is later installed via the GC Existing Database Option. You need to decide whether this database will be single-instance or RAC; Chapter 2 provides a clear basis for this decision. If you haven’t decided, do so now by revisiting Chapter 2, as the installation procedures for singleinstance vs. RAC databases vary considerably. To pre create a database to house the OMR, consult the official Oracle Database installation documentation, supplemented in this section with instructions to ensure the database supports the OMR, to streamline the installation process, and to guarantee compliance with 10g database best practices. It is outside the scope of this book to provide instructions on how to build a single-instance or RAC database for each operating system and database version certified with the Repository. The procedure differs considerably by platform, predominantly in the preinstallation steps. Other installation steps are more generic; but none of them are GC-specific, so repeating them here would add little value to understanding Grid Control. However, this section does cover all GC-specific requirements for the OMR database. The section stands on its own, e.g., you don’t have to rely upon the Oracle Enterprise Manager documentation set.21 It’s enough that you have to consult the Oracle Database documentation, informed by the content in this section. Tip Though I cannot cover Oracle Database requirements for each database version and platform, OracleMetaLink Note 401705.1 contains a comprehensive reference list of MetaLink articles detailing OS requirements for all Linux x86 and x86-64 platforms. I would be remiss if I didn’t at least identify the specific Oracle database manuals for you to reference. The Oracle Database platform-specific installation documentation, available on OTN, for single-instance and RAC databases, is as follows: ■■ If you’re installing a single-instance database, complete the preinstallation, installation and post installation tasks listed in the Oracle Database Installation Guide or Quick Installation Guide for your platform and database release. ■■ If you’re installing a RAC database, follow the tasks on each RAC node as outlined in the Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide22 for your platform and release. You generally need to complete Parts I, II, and III, which contain the following material: ■■ Part I ■■ Part II ■■ Part III Installation Planning and Requirements Preinstallation Procedures Installation and Post installation 20 Here, I am speaking of creating a new database to use with the GC Existing Database Option, as opposed to the GC installer creating a new database using the GC New Database Option. 21 Some of the database installation and configuration points in this chapter differ from or do not appear in the Oracle EM documentation, but are found in OracleMetaLink notes or based upon experience in the field. 22 This is a single Oracle guide, though the length of its title may suggest otherwise. In the Books tab of the Oracle Database Documentation Library, these books are listed with an even longer title of Oracle Clusterware and Oracle Real Application Clusters Installation and Configuration Guide for each platform. Chapter 3: Grid Control Installation 105 Steps to Install the OMR Database To put a Grid Control spin on this Oracle documentation as it relates to the Repository database, let’s divide the OMR database installation into four main steps to clarify what choices to make (steps apply to both single-instance and RAC unless otherwise noted): ■■ Install Oracle Clusterware software (RAC only) ■■ Install Oracle ASM software (10g only) ■■ Install Oracle Database software ■■ Create the Oracle Database Notice that you will install three separate Oracle homes if using RAC, or two homes if not using RAC: the Clusterware home (RAC only), the ASM home, and the database software home. Tip I strongly suggest installing the latest certified 10gR2 database Patch Set for all Oracle homes to take advantage of the new 10.2 features and latest Patch Set bug fixes. While you can run the Repository in an Oracle 9.2 database, there should be no reason to do so. You should be able to use a 10g database as long as you can dedicate a database to the OMR (i.e., the database doesn’t also need to support a thirdparty application not certified on Oracle 10g). Nevertheless, if you use an Oracle 9.2 database for the OMR, the syntax for most of the commands and initialization parameters in this section still apply. Following are pointers on how to navigate the four installation steps above, all of which are covered generically in the Oracle database documentation. For illustrative purposes in this discussion, it is assumed you are installing Oracle 10g Release 2 (10.2.0.3) for all Oracle homes, which was the latest 10.2 Patch Set certified for the Repository at the time this book was published. Be sure to check the Grid Control Certification Matrix on OracleMetaLink for the up-to-date certification information. Install Clusterware Software (RAC Only) To install a RAC database for the Repository, begin by installing the base Clusterware 10.2.0.1 release according to the Oracle Clusterware and Oracle Real Application Clusters and Configuration Guide for your particular platform. Then apply the latest Clusterware Patch Set for your platform that is certified for the OMR (currently 10.2.0.3) by consulting the release notes that accompany that Patch Set. There is not much else to say about the Clusterware with respect to supporting the GC Repository database. It is the only truly generic installation step with respect to Grid Control of the four steps covered here. Install ASM Software (10g Only) Rather than using cooked file systems for your Repository database files, consider configuring disks for use with Oracle Automatic Storage Management (ASM), which is an Oracle 10g best practice for both single-instance and RAC database storage. ASM is an integrated file system and volume manager for Oracle datafiles, spfiles, control files, online redo logs, archive logs, change tracking files, Data Guard configuration files, flashback logs, Data Pump dump files, and RMAN 106 Oracle Enterprise Manager 10g Grid Control Implementation Guide backups. ASM provides optimized I/O performance comparable to that of raw devices coupled with the easy management of a file system. The overall steps to install ASM 10.2.0.1 are to run the 10.2 database installer, choose Configure ASM, then patch ASM to 10.2.0.3. It is best practice to install a separate ASM home than that for the Oracle Database software. Among other reasons, this allows you to patch the ASM and the database software homes independently. Caution Make sure the Oracle Clusterware is running on all nodes before launching the OUI to install ASM. This enables you to perform a Cluster Installation of ASM on all cluster nodes. The ASM home is identical to the database home. As such, you install it by running the Oracle Server 10.2 OUI. Here are a few key installation points to install the ASM home and configure the ASM instance to support the Repository database (screen titles are the section subheadings): Installation Type Select Custom. Available Product Components Select the following components: ■■ Oracle Database 10g ■■ Enterprise Edition Options ■■ Oracle Advanced Security (optional)23 ■■ Oracle Net Services (Oracle Net Listener) ■■ Oracle Call Interface Create Database Select Configure Automatic Storage Management (ASM). Oracle Net Configuration Assistant Check “Perform typical configuration” to create an ASM listener on each node using default port 1521 and prefix LISTENER. Before continuing, shut down any processes, such as other listeners using port 1521. Create ASM Instance An ASM best practice is to choose “Create server parameter file (SPFILE)” rather than creating an IFILE, and for Server Parameter Filename, specify a shared raw device (15M is sufficient). ASM Disk Groups Create separate DATA and FLASH disk groups (these are arbitrary names which I use in the remainder of the chapter) for the database datafiles and Flash Recovery Area (FRA), respectively. These disk groups must be sufficiently large—taking into account the desired redundancy or mirroring level—to support the hardware operating requirements listed in Chapter 2. When the 10.2.0.1 ASM installation finishes, apply the latest database Patch Set (currently 10.2.0.3) to the ASM home. See OracleMetaLink Note 370915.1 for an excellent FAQ on ASM, including useful ASM commands and debugging tips. 23 Oracle Advanced Security (OAS) is an optional feature that must be licensed separately. To evaluate whether you need OAS for GC, see “EM Framework Security” in Chapter 16. Chapter 3: Grid Control Installation 107 Install Database Software Whether using ASM or not, it is recommended that you install the latest Oracle 10g Database software certified for the Repository (currently the 10.2.0.3 Patch Set) in a separate Oracle home from the ASM home. You can streamline the installation process by patching the database software before creating the database itself, which is the subject of the next heading. This installation order lets you circumvent the database post-installation steps for individual components—standard advice for building any Oracle Database. For those less experienced DBAs, the way this plays out is to use the OUI to install the base database software distribution (10.2.0.1) but decline to create the database, exit the installer, patch the software to the latest 10.2 Database Patch Set, then run the Database Configuration Assistant (DBCA) in standalone mode to create the database. Make the specific choices below when installing the Oracle Database 10.2.0.1 software (screen titles are the section subheadings): Installation Method (Windows Only) Select Advanced. Installation Type Select Custom. Specify Home Details Enter the following for Destination Name and Path to parallel those the Grid Control installer uses for the OMS and Agent.24 ■■ Name Specify db10g. ■■ Path Enter a path of the form /app/oracle/product/10.2.0/em10g/db10g, such as /u01/app/oracle/product/10.2.0/em10g/db10g. Specify Hardware Cluster Installation Mode (RAC database only). Select “Cluster Installation” (rather than Local Installation) and check all Node Names. Available Product Components Choose the following product components: ■■ Oracle Advanced Security (optional)25 Select. This allows you to secure the OMR database at a later time. For details on configuring OAS, see the section “Secure the Repository Database” in Chapter 16. ■■ Oracle Partitioning Select. This is a requirement for the OMR, which creates nearly 1,000 partitioned tables and roughly 700 indexes in the SYSMAN schema for storing management metrics. ■■ Oracle Enterprise Manager Console DB De select. This is Database Control, which you do not want to select for the OMR database; Grid Control will manage this and all other target databases. If you do not de select this component, the later Grid Control installation will fail. DB Control installs configuration files and Repository objects using Enterprise Manager Configuration Assistant (EMCA). Grid Control checks that neither exists because it needs to install its own version of these configuration files and Repository objects in the same locations. 24 The home names and lowest path that the Grid Control installer uses are “oms10g ” for the OMS and “agent10g ” for the Agent. 25 Oracle Advanced Security (OAS) is an optional feature that must be licensed separately. 108 Oracle Enterprise Manager 10g Grid Control Implementation Guide ■■ i SQL*Plus Select. This provides iSQL*Plus access in Grid Control to the OMR database and all target databases (configured in Chapter 4). Be sure to choose the iSQL*Plus component now because you can’t install it after patching the Database software. ■■ Other Components Select Oracle Spatial, Oracle OLAP, Data Mining Scoring Engine, and Oracle XML Development Kit. The GC New Database Option installs these components with its database software. Although the OMR does not actually rely on these components, I recommend selecting them to standardize all your GC installations irrespective of the installation type you select. Create Database Choose “Install database Software only.” First upgrade to the latest database Patch Set (done next) before creating the database. When the 10.2.0.1 OUI installation completes, apply the latest 10.2 database Patch Set to the database home. This is the same Patch Set you applied to the ASM home as described above. Create the Oracle Database At this point, you have installed the Clusterware (if using RAC), the ASM home and ASM instance, and the Oracle Database home, and all homes are at the latest available Patch Set level. The last step is to actually create a RAC or single-instance database for the Repository by running the Database Configuration Assistant (DBCA), a GUI tool that guides you through the database creation process. At the end of the Oracle Database software installation, I instructed you not to create a database, which would have launched DBCA. Instead, I advised you to install the database software only so that you could apply the latest database Patch Set before creating the database. Now that the database software is patched, you are ready to launch DBCA as a standalone tool. For specifics on running DBCA in standalone mode to create a RAC database, see Chapter 6 of the Oracle Database Oracle Clusterware and Oracle Real Application Clusters Installation Guide for your platform. To create a single-instance database for the OMR using DBCA, consult Chapter 2 of the guide called Oracle Database 2 Day DBA 10g Release 2 (10.2). Once you kick off DBCA to create either a RAC or single-instance database, you can also get online help by clicking Help on any DBCA installation screen. Tip Don’t forget to start the Clusterware, node apps, and the ASM instance on all cluster nodes before launching DBCA to create a RAC database for the OMR. Similarly, before launching DBCA to create a single-instance database, start ASM, if used. For a single-instance database, ASM, in turn, requires Cluster Synchronization Services (CSS), which are automatically installed with ASM and started as root using “crsctl start css.” Follow the documentation cited above to run DBCA in standalone mode on the OMR node, or on one of the cluster nodes, informed by the following installation choices (screen titles are the section subheadings and installation options are capitalized): Welcome Choose Oracle Real Application Clusters Database or Single-Instance Database, as already decided. Operations Choose Create a Database. Node Selection Choose all cluster nodes. Chapter 3: Grid Control Installation 109 Database Templates Select the Custom Database template. Fine-Grained Access Control (FGAC) Select FGAC, which is required to create the OMR. Management Options De select Configure the Database with Enterprise Manager. This option is intended for installing a target database to point to an existing Grid Control installation. Here, by contrast, you are creating a database to host the OMR component for a new GC installation, which, in its course, automatically configures the OMR Database as a Grid Control target. Storage Options (10g Only) Select Automatic Storage Management (ASM) if it’s installed.26 As already mentioned, using ASM as the storage mechanism is a database best practice for both single-instance and RAC databases. If you select ASM, the next installation screen allows you to choose the DATA disk group created earlier as the database storage location. Database File Locations Select Use Oracle-Managed Files.27 OMF is a database best practice for the generic database datafiles. Unfortunately, the GC OUI does not allow you to specify an OMF file format for the GC tablespaces it creates. For the Database Area, enter the DATA disk group as “+DATA”. Recovery Configuration Note the following about the Recovery Configuration options: ■■ Specify Flash Recovery Area (10g only) Check this option as a database best practice. If using ASM, specify the FLASH disk group created earlier as the FRA and allocate a FRA size of two, preferably three times the database size. (After installing Grid Control, which creates two tablespaces, the database size will be approximately 2.5GB.) ■■ Enable Archiving Do not check this box. It will speed up the Grid Control installation process. Instead, for recovery purposes you can take a cold backup of the database once it’s built, as recommended later in this chapter. Chapter 4 reminds you to enable archiving at the end of the OMR database configuration process. Database Content Under the Database Components tab, I recommend selecting the same components that the GC New Database Option installs, which are Oracle Data Mining, Oracle Text, Oracle OLAP, and Oracle Spatial. Also, click the Standard Database Components button and select all three components: Oracle JVM, Oracle XML DB, and Oracle Intermedia. Components not required and which should be grayed out (i.e., nonselectable) anyway are Oracle Ultra Search, Oracle Label Security,28 Sample Schemas, and Enterprise Manager Repository. The only component that you must be careful not to select is the Enterprise Manager Repository, as the GC Existing Database Option must install the Repository. Initialization Parameters It is easier to wait until after DBCA creates the database to change the bulk of the initialization parameters the OMR requires or recommends. This is because the DBCA user interface (UI) is cumbersome, whereas, once the database is created, you can change 26 The other two storage choices for RAC databases are Raw Devices or Cluster File System, and choices for singleinstance databases are Raw Devices or File System. 27 The other option is to Use Common Location for all Database Files, which is a non-OMF selection. 28 The Oracle EM 10.1 documentation incorrectly listed Oracle Label Security as a requirement, but it is not required either for EM 10.1 or 10.2. 110 Oracle Enterprise Manager 10g Grid Control Implementation Guide all initialization parameters more quickly by scripting the process, for example. A few parameters, however, should be set within DBCA, as they are difficult to change after DBCA creates the database. These parameters are as follows (bolded text indicates tab names on the Initialization Parameters screen): ■■ Sizing Ensure Block Size is set to 8192 bytes, the Oracle-recommended value for an OMR database. Set DB_BLOCK_SIZE now at database creation time, because changing it requires recreating the database. ■■ Character Sets Make sure the chosen database and national character sets store the main language used by GC administrators and additional language groups desired. A good choice is the Database Character Set Use Unicode (AL32UTF8) and the accompanying National Character Set “AL16UTF16 - Unicode UTF-16 Universal character set.”29 Also, select your desired Default Language and Default Date Format. ■■ Database Storage It’s easier to specify the correct redo log sizes for DBCA to create than it is to re create these redo logs afterward. Redo log sizing depends upon the anticipated size of your GC environment. For redo log recommendations, see Table B-2 in Appendix B (online at www.oraclepressbooks.com). At this time, don’t worry about the number of redo log groups, the SGA size, or tablespace, datafile, or tempfile specifications. In the next section, “Configure/Tune a Database for the OMR—Part 1 of 2,” we use SQL*Plus to change them as required for Grid Control. Creation Options The following provides information about the Creation Options: ■■ Create Database Check this box, particularly for a RAC database. If you don’t check this box to let DBCA create a RAC database in the GUI, and rely instead on database creation scripts (see next bullet point), then you will need to perform additional manual steps not documented here or in the Oracle documentation to complete the RAC installation process. ■■ Generate Database Creation Scripts I recommend selecting this option. If a singleinstance database installation fails, you can run the scripts (rather than re running DBCA and re entering all required information), or create an identical GC environment on another host, if needed. The scripts also serve as a record of the installation process.30 This concludes the database installation process for a single-instance or RAC database to house an OMR installed using the GC Existing Database Option. Configure/Tune a Database for the OMR—Part 1 of 2 Perform the steps in the section “Configure / Tune a Database for the OMR—Part 1 of 2” in Appendix C (online), which lists database configuration and tuning tasks to complete before launching the installer and selecting the GC Existing Database Option. As I’ve already mentioned, you 29 These are both Unicode character sets, which store Western and non-Western characters. See Oracle Database Administrator's Guide 10g Release 2 (10.2), Manually Creating an Oracle Database, Considerations Before Creating the Database, Table 2-1. 30 There is a bug in the init.ora script created by selecting Generate Database Creation Scripts. Trying to create a 10.2 database with these scripts produces the error “ORA-01678: parameter db_file_name_convert must be pairs of pattern and replacement strings.” Resolve this error by unsetting its value. Chapter 3: Grid Control Installation 111 could have performed most of the tuning steps (such as the initialization parameter changes) when running the Database Configuration Assistant (DBCA). However, it’s quicker to do all configuration and tuning through SQL*Plus once you’ve created the database to house the OMR. After you run the Grid Control installer as described in the next section, and patch Grid Control according to the instructions in Chapter 4, I will refer you back to the online Appendix C to complete the steps in the section “Configure / Tune a Database for the OMR—Part 2 of 2.” The reason for the two parts is that, by and large, the steps in Part 2 (such as enabling ARCHIVELOG mode), if implemented before installing and patching Grid Control, would reduce the performance of the Grid Control installation and patching process. Run the Installer—GC Existing Database Option Like the GC New Database Option, the GC Existing Database Option installs an OMS, OMR, and chain-installed OMA, but with two differences. The principal difference is that the GC Existing Database Option doesn’t create a database for the OMR but installs the OMR into an existing qualified database. The second distinction is that the GC Existing Database Option can install an OMR, OMS, and chain-installed OMA on the same host, or it can install the OMR on a separate host from the OMS and chain-installed OMA. (The GC New Database Option can only install GC components on the same host.) When you choose the GC Existing Database Option, the installer creates the OMR in an existing single-instance or RAC database you point to, and installs the OMS and chain-installed OMA on the local host. The GC installer is not what you would call “RAC-savvy” (as I will explain shortly), but you can circumvent the few RAC-related gotchas by following the instructions in this section. Tip Now would be a good time to take a cold backup of the existing database, particularly if you did not enable archiving yet, which I recommended to keep the GC installation time to a minimum. A backup will preserve the database creation, configuration, and tuning you just completed, providing a way to recover and restore to this point should the Grid Control installer fail or other unrecoverable error occur. Let’s kick off the Grid Control installer now (for instructions see the section above, “Starting and Monitoring the Installation”). The GC Existing Database Option is very similar to the GC New Database Option. The obvious difference is that on the Specify Installation Type screen, you select Enterprise Manager 10g Grid Control Using an Existing Database. Note To anticipate all screen shots you will encounter, see earlier Table 3-1, which lists the screens that appear when you choose the GC Existing Database Option (column 2), and which references the figure number of the screen (column 9). Each screen is also labeled as to which Grid Control installation type(s) render it (the label is “Existing” for the GC Existing Database Option). 112 Oracle Enterprise Manager 10g Grid Control Implementation Guide The other differences are fewer screens because some only apply to the GC New Database Option and two additional installation screens for the Existing Database option. One screen shot requests that you enter a password for the new SYSMAN user, which is simple enough and not shown here. The other screen shot is entitled Specify Repository Database Configuration (see Figure 3-14). This screen can trip you up. Let’s review the information it requests, field by field, observing the following guidelines: ■■ Database Hostname Use the fully qualified hostname of the existing database. If you’re using a RAC Repository, enter one of the node names. ■■ Port Enter the database listener port (the default port is 1521). ■■ Service/SID ■■ For single-instance OMR, enter the DB_NAME. ■■ For a RAC Repository, rather than the DB_NAME, specify the INSTANCE_NAME running on the node entered above for Database Hostname. (After the Grid Control installation is complete, as Chapter 4 instructs, you will need to modify the Management Service connection string from INSTANCE_NAME to DB_NAME to take advantage of client failover to the other RAC instance in the event of a RAC host outage.) The example INSTANCE_NAME in the screen shot is emrepa1 to exemplify a GC environment with multiple sites. The DB_NAME for this site is emrepa and for additional sites would be emrepb, emrepc, etc. ■■ SYS Password Enter the current SYS password. FIGURE 3-14. Specify Repository Database Configuration: Existing, Add’l Chapter 3: Grid Control Installation 113 ■■ Tablespace Locations Specify the full path for the datafiles. You cannot specify OracleManaged Files (OMF) for these GC tablespaces (though you could convert them to OMF post-installation, if desired). So if you’re using ASM, when prompted for the GC tablespace locations, specify a non-OMF string with the full path for the datafiles. It is recommended that you use the default datafile names, in which case the full path when using ASM would be of the form +DATA/ /datafile/ . For example, a good naming convention for these datafiles is as follows: Management Tablespace Location: +DATA/emrep/datafile/mgmt.dbf Configuration Data Tablespace Location: +DATA/emrep/datafile/mgmt_ecm_depot1.dbf When you install the Management Repository into an existing database (including a RAC database), the installer does not allow you to set the size of these required EM tablespaces. The default initial datafile extents assigned are not usually sufficient for a production environment, so the installer sets AUTOEXTEND ON to allow these datafiles to grow. However, these default extent sizes become a problem when storage for the database (usually a RAC database) is on a raw device, because the AUTOEXTEND ON feature does not work for raw devices. If the database used for the Management Repository is configured with raw devices, there are two options for increasing the size of the Repository for each of the two EM tablespaces: ■■ Pre create multiple raw partitions, setting the first partition size equal to the default size of the tablespace as defined by the installation process. The second partition allows for growth. ■■ Alternatively, pre create a large partition sized to allow for EM growth, create a tablespace using the default size, create a dummy object that will increase the size of the tablespace to the end of the raw partition, and then drop the dummy object. Regardless of which option you choose, when using raw devices disable the default AUTOEXTEND ON space management for all objects. (Do not disable AUTOEXTEND ON when using ASM-managed storage for the Repository database.) Trick to Installing the OMS on a RAC Node It is unsupported to install an OMS on a RAC node prior to EM10g Release 10.2.0.2. However, you must still know the trick to installing a 10.2.0.3 or 10.2.0.4 OMS on a RAC node. On the Specify Repository Database Configuration screen; if you specify the local INSTANCE_NAME or local node where running the GC installer, you get the following error due to Bug 4745547: “You do not have sufficient privileges to write to the specified path for tablespaces.” The workaround is to specify a remote RAC node name and INSTANCE_NAME. The error is due to ASM, not RAC. If the OMS and the Repository Database are on the same box, then Oracle tries to check the path writability of the tablespace locations entered. For ASM disks, the path writability returns false (patch writability cannot be used for ASM disks), the installation fails at this validation, and it cannot proceed. The reason for the success of this workaround is that this check is not performed if the database is perceived to be on a remote node. The only anomaly with this configuration is that some of the OMS ports the GC installer selects are nondefault. 114 Oracle Enterprise Manager 10g Grid Control Implementation Guide Install Standalone Agents on Dedicated OMR Nodes You don’t need to explicitly install an Agent on the OMS host. The Grid Control installation does this for you with the chain-installed Agent, regardless of the option selected—either using the GC New or Existing Database Options, or when installing an Additional Management Service. By contrast, you do need to install a standalone Agent on an OMR host when it is remote to an OMS installed using the GC Existing Database Option. This option lays down an OMS and chain-installed OMA on the local server, and creates the Repository through a SQL*Net connection in an existing single-instance or RAC database on a local or remote node(s). But the GC Existing Database Option does not install a Management Agent on each remote OMR node; you must install an Agent on each OMR node in a separate process, using any standalone installation method. See Chapters 5, 6, and 7 for instructions on installing a standalone Agent via one of six available installation methods. Note You don’t have to install standalone Agents right now on dedicated OMR node(s). You can wait until you begin installing standalone Agents in Chapter 6. Whenever it is, I suggest that you deploy the first standalone Agents on OMR nodes. Monitoring the OMR database and node(s) is paramount, as all other target monitoring relies on the OMR database remaining operational. Log in to the Web Console Now that you’ve installed Grid Control using either the GC New Database Option or GC Existing Database Option, you probably want to log in to the Console before proceeding further, just to make sure the GC installation was successful, or at least successful enough for you to get in the front door, so to speak. If the GC installation was successful, all GC processes required for login will automatically start, allowing you immediate access to the Console as the built-in SYSMAN administrator, and with the password specified during the installation. The following table lists the URL variations to log in to the Grid Control Console.31 32 31 Login Method Protocol Port for UNIX Port for Windows Used OMS Host OMS Host Grid Control Console Login URL Web Cache HTTP Web Cache 7777 80 http:// : /em HTTPS (must configure) (must configure) https:// : /em HTTP Server HTTP 4889 4889 http:// : /em HTTP Server HTTPS 1159 1159 https:// : /em 32 If you used the static ports feature, your port numbers may be different. This Web Cache HTTPS URL does not work out-of-box after you install Grid Control. See “Configure/Enforce HTTPS Between Browser and Web Cache” in Chapter 16 for details on how to configure this access. By default, upon installation, traffic between browser and Console via Oracle HTTP Server is already secured over HTTPS on port 1159. 32 Chapter 3: Grid Control Installation 115 As you see, Console login is via OracleAS Web Cache or the Oracle HTTP Server (OHS), over a secure SSL (HTTPS) or nonsecure (HTTP) connection. You can log in to the GC Console via Web Cache, which in turn accesses OHS in the background, or you can bypass Web Cache and log in directly via OHS. It is recommended that you log in via Web Cache (either over HTTP or HTTPS as your security requirements dictate) because page performance is usually slightly better. But it’s useful to know how to bypass Web Cache, such as when you’re trying to isolate a Console login problem. The URL protocol varies (HTTP or HTTPS) depending on whether you use SSL encryption for browser <-> Console communications. The URL port also differs by platform (UNIX or Windows) for Web Cache login. Console access via Web Cache doesn’t help performance much when compared with that via OHS because the information provided in Grid Control is largely dynamic. Web Cache does cache some information that doesn’t change (such as icons, headers, and footers), but this is a small percentage (typically 1 percent to 5 percent) of the data returned to the Console. (You can sometimes distinguish the cached data by the way it sometimes loads before the rest of a page.) The Console must display target information obtained in real-time for accuracy, so this information cannot be cached. Therefore, you receive only a slight benefit to caching Grid Control data in Web Cache.33 I still prefer to log in via Web Cache because any performance improvement, however little, is better than nothing. Also, Web Cache is used to monitor the built-in EM Website service (see Chapter 12). For now, ensure that you can log in as SYSMAN via all configured methods shown in the table above. Once logged in, you’re in a whole new world of IT management, but the goal here is just to make sure things appear in order. If you can see all the OMS subcomponents on the Targets tab under the All Targets subtab, and if most of these targets are showing the correct UP status (a green arrow pointing up), then this is a good enough spot check for now.34 However, knock yourself out and tool around in the Console until you’re satisfied the Console is running smoothly. Don’t get too carried away, though. From Chapter 10 forward, we spend most of our time in the Console configuring Grid Control. Install the Grid Control Security Certificate in Your Browser Each time you log in to the Console using HTTPS, a security alert dialog box appears in your browser informing you that the certificate of the Grid Control site is not trusted. Your browser is simply letting you know that it does not accept the Grid Control certificate for the purposes of identifying the Console web site itself. In other words, your browser does not assume on your behalf that the Grid Control application certificate authority (CA) root certificate is trustworthy. This root certificate is a trusted certificate included in the OHS wallet,35 which also contains the OMS server’s certificate and private key. Your browser prompts you to set up a trusted relationship with this CA root certificate as required by Secure Sockets Layer (SSL) to encrypt browser <-> OHS communication. 33 Several analysts in the Oracle Support EM group told me that some customers disable Web Cache to conserve OMS resources. If you ask these analysts why Oracle supplies Web Cache with Grid Control, their answer is because it’s bundled with OracleAS, which is the framework for the OMS middle tier. 34 Don’t be surprised if the database and ASM targets (except a new Management Repository database) show a DOWN status. You must configure them by entering the password for the DBSNMP and SYS users, respectively, as described in Chapter 11. 35 The OHS wallet, generated during Grid Control installation, is located in the directory $OMS_HOME/Apache/ Apache/conf/ssl.wlt/default as defined by the value for SSLWallet file: in $OMS_HOME/Apache/Apache/conf/ ssl.conf. For more information on this wallet, see “Configure/Enforce HTTPS Between Browser and Web Cache” in Chapter 16. 116 Oracle Enterprise Manager 10g Grid Control Implementation Guide Appendix I contains the procedures to install the Grid Control certificate in Internet Explorer as well as to suppress the Security Information dialog box. The procedure to install the certificate for other certified browsers (including Netscape, Mozilla, Firefox, and Safari) is different, but the basic idea is the same: to install and accept the certificate forever. (See your browser documentation for the certificate installation procedure.) After installing the certificate for most browsers, you won’t receive any further security alerts when logging in to the Console on your client workstation using this browser. However, I’ve found that in Netscape Navigator a dialog box called Security Error: Domain Name Mismatch appears every time, even after accepting the certificate forever, which is particularly bothersome. Tip I recommend using Microsoft Internet Explorer for the Web Console, as it allows you to use web application features without additional configuration required by other browsers. These features include service tests and the Transaction Recorder. (See “How to Configure a Service” in Chapter 12 for details on these features.) Install an Additional OMS As the name implies, this option installs a second (third, fourth…) OMS and chain-installed Agent on the local host where running the GC installer. The prerequisite for choosing this option is to first install an OMR, OMS, and chain-installed OMA, i.e., first run the GC installer and choose either the GC New Database Option or the GC Existing Database Option. The option you choose dictates where you locate an additional OMS and chain-installed Agent, as follows: ■■ For either option, install an OMS and chain-installed Agent on a system not running any other Grid Control components, or ■■ Install an OMS and chain-installed OMA on a node running an existing single-instance or RAC36 Repository database, where the primary OMS was installed on a separate host. This is not a likely scenario.37 The procedures for the GC Additional OMS Option and GC Existing Database Option are very similar. The only difference is that the former only updates the SYSMAN schema to reflect that another OMS exists, whereas the latter creates the Repository in an existing database in the SYSMAN schema. OMSs in an Active/Active configuration are independent of each other, except as relates to the Shared Filesystem Loader (if configured).38 However, do not mistake independence for ignorance. As fellow members of the GC infrastructure, OMSs are aware of each other in that they all appear as targets in the GC Console. This “independent awareness” has its benefits. For instance, you can shut down OMS #1 for maintenance and administrators can access the Console via OMS #2. 36 Again, Oracle does not support installing an OMS on a host running Clusterware until GC 10.2.0.2. In this scenario, you would install the first OMS on a different host than the OMR, then install a second OMS on the same host as the OMR. It is more likely that you would do the reverse, i.e., install the first OMS on the same host as the OMR (using the GC New Database Option), then install a second OMS on a different host than the OMR (via the GC Additional OMS Option). 38 See Chapter 15 for more information on both the Shared Filesystem Loader and OMS Active-Passive configurations. 37 Chapter 3: Grid Control Installation 117 Note The screen shots above with the label “Add’l” appear when choosing the GC Additional OMS Option. For a complete list of screen shots for this option, see column 3 of Table 3-1. As contrasted with the GC New or Existing Database Options which install the initial OMS, the following screens do not appear when installing an Additional Management Service: ■■ Specify Repository Database Configuration The installer does not prompt you for GC tablespace information because the other options create the required tablespaces. ■■ Specify Optional Configuration The installer does not prompt for e-mail notification or OracleMetaLink credentials because you enter them when installing Grid Control using the other options. As a general guideline, the number of required Repository processes increases by 100 for each additional OMS. My recommended setting of 2000 for PROCESSES (see Appendix C, online) should be sufficient for multiple (i.e., three or four) OMSs. Tip When installing an additional OMS, use the static ports feature to ensure that the port configuration on all OMS servers is identical (see the section “Static Ports Feature” earlier in the chapter). Perplexingly, Grid Control does not use all the same default ports for additional OMSs, even if these ports are available. Summary This chapter focused on installing the Grid Control infrastructure using the GC New or Existing Database Options, and on installing additional Management Services (beyond the first) using the GC Additional OMS Option. The chapter began with gathering installation information, addressing installation bugs, initializing the OS environment for the GC installer, and running the GC installer with any desired options, silently or using the static ports feature. In the section on the GC New Database Option, I presented screen shots and instructions to complete them. Following this section is a major section on the GC Existing Database Option devoted to creating, configuring, and tuning a new database to house the Repository. In this section, you learned which screen shots are unique to the GC Existing Database Option (not presented by the GC New Database Option), and the corresponding input required. I concluded the chapter by providing quidance when choosing the GC Additional OMS Option. Because this procedure is almost the same (only with several fewer screen shots) as the one for the GC Existing Database Option, it suffices to point out which screen shots do not appear. This page intentionally left blank 4 Grid Control Post Installation 119 120 Oracle Enterprise Manager 10g Grid Control Implementation Guide I n Chapter 3 you installed the latest Grid Control Full Installer release for your platform, including the Oracle Management Repository (OMR), one or more Oracle Management Services (OMSs), and tag along Oracle Management Agents (OMAs). In this chapter, you will perform the following post installation tasks: ■■ Patch the Grid Control environment ■■ Configure the OMS host(s) ■■ Configure and tune the OMR node(s) and OMR database See the beginning of each section for a summary of the specific tasks to accomplish in that section. Let’s begin with the first task: patching Grid Control to bring it up to the latest patch level. Patch Grid Control Following are the four categories of Grid Control patches presented in this section, some of which you may already have applied: ■■ Apply the latest Grid Control Patch Set ■■ Apply the latest Database Patch Set certified for the OMR ■■ Apply the latest EM Critical Patch Update (CPU) ■■ Apply required Grid Control one-off patches Note Most Grid Control patching functionality requires one of the OEM Provisioning Packs. For more information, see Enterprise Manager Licensing Information, available on OTN in the Reference section of the EM Documentation Library. Grid Control can stage patches for itself, and in some cases can actually patch itself. The limitation is that Grid Control can only apply patches that rely upon the opatch executable, which updates the OUI inventory for the product being patched. As for the patches covered here, it’s efficient to stage almost all of them in the Console, and I illustrate the navigation path to do so. This way, you learn how to use Grid Control while configuring Grid Control. I also point out how you can apply one of these patches (the CPU) if running multiple OMS hosts. It gets a little reflexive here, but think of it this way: Grid Control can patch itself like a doctor can patch (or “operate” on) himself or herself. A doctor can stitch his or her own leg and even perform minor surgery under local anesthesia, but certainly can’t operate on himself or herself under general anesthesia. For Grid Control, local anesthesia to self-patch is provided by multiple OMSs. You keep one OMS running while patching another (check first that the patch Readme allows for this). Patching an OMR database with Grid Control is almost out of the question, since even a one-off database patch through opatch requires shutting down the database, which shuts Chapter 4: Grid Control Post Installation 121 down Grid Control. The upshot is that the OMR database and at least one OMS must remain running when applying a patch through the Console. Caution It is not supported to change the Application Server configuration forming the Grid Control middle-tier outside of applying Grid Control patches to it. Let’s begin the Grid Control patching process now. It’s always best to test these patches in a nonproduction environment before putting them into production. However, that ship sailed at the beginning of Chapter 2 when you decided how many GC environments to build: ■■ If you built a production GC environment to manage both production and nonproduction targets, then take a good backup (as noted below) and hope for the best. ■■ If you opted to build both a production and nonproduction environment, then you've already installed the nonproduction environment. If the production hardware is ready, consider starting the production build now so as to begin production monitoring as soon as possible. To do so, double back and complete to the instructions in Chapters 2 and 3 on the production hardware. Then you can return to this chapter and patch nonproduction, with the assurance that these patches will not affect production. Apply the Latest Grid Control Patch Set Regardless of whether you used the GC New or Existing Database Option to install Grid Control, you should download and apply the latest release of the GC Patch Installer1 for your platform (currently 10.2.0.3 or 10.2.0.4, but referred to here simply as the “GC Patch Set”). You must be on GC release 10.2.0.1 or higher to apply the latest GC 10.2.0.x Patch Set. If you were lucky enough when installing Grid Control to be on an OMS platform for which the latest GC release was available in a Full Installer version, then you can skip this section. GC Patch Sets are cumulative. For instance, you can directly apply the GC 10.2.0.4 Patch Set without first applying the 10.2.0.3 Patch Set. The latest GC Patch Set fixes many bugs in the base 10.2.0.1 release that provide critical functionality. GC Patch Sets also introduce new functionality and add-ons. One new feature as of the GC 10.2.0.2 Patch Set is Oracle Configuration Manager (OCM), a separate product that extracts GC configuration information and uploads it to Oracle Support. (OCM makes opening Service Requests easier because you don't need to manually supply GC configuration details.) Tip You can apply the GC Patch Set to many Agents at one time through the Grid Control Console. For details, see the GC Release Notes for your platform. By contrast, you cannot patch OMS hosts through the Console, as the Patch Set requires shutting down all OMS hosts to patch any one of them. 1 A GC Patch Installer release contains an OMS, OMA, and Management Packs, but no OMR or OMR Database. 122 Oracle Enterprise Manager 10g Grid Control Implementation Guide Let me summarize the GC OMS patching process detailed in the Release Notes to clarify and supplement them in a few places. (I do not go into detail on applying the GC Patch Set to Agents, as the Release Notes are very clear on this subject.)2 1. Download the Patch Set software and Readme for your OMS platform to a shared location amongst OMS hosts, if possible. You may have already done so, as recommended in Chapter 1. (While downloading, you can proceed to the next step.) The Readme.txt file is located under the doc directory in the software distribution. You can also download the Readme separately for certain platforms from the OTN Enterprise Manager Downloads page at http://www.oracle.com/technology/software/products/oem/index.html under the Patch Installers section via the README link. See the Readme for general GC Patch Set directions and consult the Release Notes included with the Patch Set for specific instructions. 2. Take a cold backup of your entire GC installation, preferably via server-wide backup. This includes all OMS homes, the OMR Database, all OMR homes, and any standalone Agents already installed, (While backing up, you can go to the next step, which requires some reading.) The most reliable mechanism for de installing GC Patch Sets is to restore from backup. See “Backups Before and After a Configuration Change” in Chapter 14 for details. 3. Make a Go/No-Go decision now on whether to apply the Patch Set. Read the Known Issues3 section of the GC Patch Set Release Notes to determine whether any bugs are showstoppers. You may decide that the bugs are such that you’d rather wait for the next GC Patch Set, though this is unlikely. (You won't have the option of waiting if applying GC 10.2.0.4, as this is the terminal 10.2 release.) These bugs were discovered even before the Patch Set was released. Generally speaking, the bugs should be fewer and less loathsome than those in the previous Patch Set. Workarounds are available for some of these bugs. Flag any “must-have” workarounds. You’ll need to implement them after applying the Patch Set (done in step 10 below of this procedure). 4. Perform tasks specified in the Preinstallation section of the Release Notes. If you already began configuring Grid Control in the Console or have patched it since initially installing it, see the Release Notes for possible preinstallation steps. However, if you have not used or patched Grid Control yet (i.e., you’re following instructions in this book), there is only one possible preinstallation task, depending on how you installed Grid Control. ■■ If you selected the GC New Database Option, or if you chose the GC Existing Database Option using Oracle Database 10.1.0.4 or 10.1.0.5, apply patch 4329444 to the OMR database. ■■ If you selected the GC Existing Database Option and used Oracle Database 10.2 for the Repository, there are no preinstallation tasks for the GC Patch Set. 5. Set a side-wide blackout to suppress alert notifications for all GC components. To set this blackout, log in to the Console with OPERATOR privilege on all targets, click the Setup link at the top right of the home page, click the Blackouts tab on the left navigation page, and click Create. On the Properties page, select “all target types” in the Type 2 Perhaps i goes without saying that you only need to patch the OMR database once, regardless of the number of OMS hosts you installed. 3 I have a less euphemistic name for the Known Issues section: New Bugs with this New Patch Set. Chapter 4: Grid Control Post Installation 123 field under Available Targets and click Move All to move all targets to Selected Targets. Advance to the Schedule page, enter immediately for the Start Date, and Indefinite for the Duration of the blackout. 6. Shut down all OMS services associated with the Repository by executing $OMS_HOME/ opmn/bin/opmnctl stopall on each OMS host. However, leave the Repository database and listener running. 7. Set ORACLE_HOME to the OMS home in the environment on each OMS host. On each OMS you intend to patch (starting with the first OMS installed), log in to an XWindows session as oracle and set the ORACLE_HOME to the OMS home in the environment. This ensures that the OUI automatically picks up the ORACLE_HOME for the OMS. (You can also pass the ORACLE_HOME as an argument to the installer executable—see the next step.) 8. Apply the GC Patch Set to each OMS host. When you apply the Patch Set to the first OMS host, it also patches the associated Repository, whether located on the local OMS host or remote to it. To launch the installer, change directories to 3731593/Disk1/ in the Patch Set distribution, and execute runInstaller on UNIX or setup.exe on Windows.4 Specify ORACLE_HOME= as an argument to the installer executable if you did not set it in the environment in the previous step. 9. Complete any applicable GC Patch Set post installation tasks for each OMS, if applicable; some of these tasks depend upon whether metadata from certain platforms (such as Solaris) is present in the OMR, and others redefine the host metrics called Generic Log File Monitoring and Log File Pattern Matched Line Count. 10. Implement any “must-have” workarounds for each OMS that you identified in Step 3 above from the Known Issues section of the GC Patch Set Release Notes. 11. Apply the Patch Set to all tag-along and standalone5 Agents (ideally in one operation) according to the instructions in the GC Patch Set Release Notes. Apply Latest Database Patch Set Certified for OMR The Oracle Management Repository is currently certified on Oracle Databases 9.2.0.6+, 10.1.0.4+, and 10.2.0.2+ (i.e., “+” means “and subsequent Patch Sets in the series”), both standalone and RAC. Actually, Database 11.1.0.6 is also certified, but I can only recommend using it with EM 11g, which is being developed on Database 11g. As I recommended in Chapter 2, you should seriously consider running Database 10.2 to take advantage of its latest features, and running the latest 10.2 Patch Set available for your platform to incorporate the most recent bug fixes. If you installed Grid Control using the GC Existing Database Option according to the instructions in Chapter 3, then you already patched the OMR database to the latest 10.2 release. However, if you selected the GC New Database Option, the GC installer laid down Oracle Database 10.1.0.4. In this case, I suggest 4 5 You can also run the Patch Set silently using a response file. Chapter 7 reminds you to do this in the “Agent Post-Installation Steps” section following standalone Agent installations. 124 Oracle Enterprise Manager 10g Grid Control Implementation Guide upgrading the OMR database and software to 10.2.0.1, then applying the latest 10.2 Patch Set for your platform. ■■ For instructions on upgrading Oracle database and software from 10.1.0.4 to 10.2.0.1, see the Oracle Database Upgrade Guide 10g Release 2 (10.2) available on the Oracle Technology Network (OTN) under the Database link below the Documentation tab. ■■ To patch the Oracle Database and software from 10.2.0.1 to the latest release for your platform, see the Oracle Database Patch Set Notes 10g Release 2 for that release and operating system. You can download the Readme and Patch Set from OracleMetaLink. While Grid Control can stage the latest Oracle Database 10.2 Patch Set, it cannot apply the Patch Set because it requires shutting down the OMR database. Apply the Latest EM Critical Patch Update Oracle highly recommends that customers apply the latest Critical Patch Update (CPU) for EM Grid Control. A CPU is a cumulative set of patches that address both critical security vulnerabilities and nonsecurity bugs on which these security fixes depend. CPUs are put out every quarter (in January, April, July, and October) and, like Patch Sets, are rigorously tested as a collection, so they are more reliable than one-off patches. To determine the CPU requirements and patch number to apply, see the Critical Patch Update Advisory on OracleMetaLink. This document contains a link to a document called the Critical Patch Update Availability for Oracle Server and Middleware Products. This document, in turn, contains two sections: ■■ Minimum Requirements specify minimum Patch Set requirements for the Oracle Application Server and the Oracle Database components of EM. These components must be on the minimum Patch Set levels stated here to apply the CPU. ■■ Oracle Enterprise Manager Patch Availability shows the CPU patch numbers to apply for the OMS/OMR and for the OMA (separate patch numbers) based on the EM release and platform (UNIX or Windows). You can apply the CPU manually or use Grid Control to stage and apply some of the CPU to certain GC components, provided you comply with all CPU Readme instructions. For example, applying the Enterprise Manager CPU to an OMS requires shutting down that OMS. Thus, to apply the CPU to a nonrunning OMS via Grid Control requires logging in to the Console through a separate, running OMS. If you have at least two OMS hosts, say OMS #1 and OMS #2, patch them as follows, beginning with OMS #1: 1. Shut down OMS #1. 2. Log in to the Console via OMS #2, and click Patch Advisory under Deployments. 3. On the Patch Advisories tab, under Interim Patches to Apply, click the CPU patch number, select the OMS #1 home under Affected Oracle Homes, and click Patch. Follow the Patch Wizard to stage and apply the patch. The only step in the Wizard that is not self-explanatory Chapter 4: Grid Control Post Installation 125 is on the Stage or Apply page. In the Pre-Patch and Post-Patch sections, you need to enter the following commands to shut down and restart OMS #1, which you are patching: %oraclehome%/opmn/bin/opmnctl stopall %oraclehome%/opmn/bin/opmnctl startall Enter the %oraclehome% variable above literally, as it represents the OMS #1 home you are patching. Complete the remaining Patch Wizard pages and submit your request. Grid Control submits a job to apply the CPU to the selected OMS #1 Oracle home. Click the View Job button for details on the job’s progress. 4. When finished, shut down OMS #2, and start and log in to the Console via the now patched OMS #1. 5. Repeat this patching procedure for OMS #2. Apply Required Grid Control One-off Patches One patching step many GC administrators overlook is applying any one-off patches or one-offs that are functionally required or needed to implement workarounds. One-offs are patches that address individual bugs, some of which resolve significant functional bugs or deficiencies. Oracle creates one-offs as interim patches to apply on top of a specific (hopefully the latest) GC Patch Set, and rolls these one-offs into the next GC Patch Set. Many of the one-offs are intended for the OMR database. The proactive way to determine which one-offs you functionally require is to scan their descriptions (and Readme files if needed), then choose those one-offs that provide fixes you cannot live without. The reactive way to choose needed one-offs is to wait until Grid Control fails in some unacceptable way, then look for a one-off that fixes that failure. Tip I recommend the reactive approach of not applying GC one-off patches unless absolutely necessary. Oracle apparently agrees with this conservative approach, as evidenced by the following statement placed in nearly every database one-off Readme: “You must have NO OTHER PATCHES installed on your Oracle Server since the latest Patch Set (or base release x.y.z if you have no Patch Sets installed).” You can identify an up-to-date list of one-off patches for a specific GC Patch Set, then download these patches directly from OracleMetaLink and stage them manually or download and stage them in Grid Control. Note See “Perform OMR Partition Maintenance” in Chapter 17 for specific GC and OMR Database one-off patches that may be required, depending on the versions you are running. Let’s take a closer look at how to download and stage one-off patches, both in OracleMetaLink and in the Console. I use the Linux x86 platform in the examples. 126 Oracle Enterprise Manager 10g Grid Control Implementation Guide Downloading One-off Patches in OracleMetaLink To download a patch in OracleMetaLink and stage it manually, follow these steps: 1. Download a one-off patch by clicking the Patches & Updates tab, then click the Advanced Search link. Complete the requested fields as shown below by clicking the Flashlight icon, selecting the appropriate values for your site, and then clicking Go. Make sure the Release field reflects your currently installed GC release. 2. Click the View Readme icon for each patch you are interested in evaluating and then click the Download Now icon to download the patch. 3. Manually unzip the patch to stage it. Chapter 4: Grid Control Post Installation 127 Downloading and Staging One-off Patches in Grid Control To download and stage any patch (including a one-off) in the GC Console, do the following: 1. Go to the Deployments tab, and click Patch Oracle Software under the Patching section. 2. The Patch Wizard leads you through the staging/patching process. On the Select Patch page, click Search Criteria. 3. Enter the required fields, and click Search. The patches matching the criteria are returned as shown above. Under Search Results, select the desired patch. If needed, click View Details (which lists a patch description, whether you’ve already downloaded it to Grid Control, etc.) or click View ReadMe. To stage the patch, click Next 128 Oracle Enterprise Manager 10g Grid Control Implementation Guide 4. On the Select Destination page, select the Destination Type called Host and Directory. Enter the Host and Directory (or multiple hosts and directories) where you want to download and stage the patch, and click Next. When you specify /home/oracle as the Directory, the patch directory structure GC creates looks like this: pg04.perftuning.com > pwd /home/oracle/EMStagedPatches/3162149 pg04.perftuning.com > ls -la total 2080 drwxr----3 oracle dba 96 drwxr----3 oracle dba 96 drwxr-xr-x 4 oracle dba 96 -rw-r----1 oracle dba 1056947 Jun Jun Mar Jun 27 27 20 27 13:24 13:23 02:05 13:23 . .. 3162149 p3162149.zip As shown above, Grid Control creates and appends the subdirectory EMStagedPatches/ to the Directory you specify, and then places the patch zip file in this subdirectory alongside another subdirectory where it unpacks the patch file. Click Next. Chapter 4: Grid Control Post Installation 129 5. Complete the remaining Patch Wizard pages, which are self-explanatory, and submit your request for hand-off to the GC job system to execute. Click the View Job button to ensure that the patch was staged successfully. The advantages of using the Console rather than OracleMetaLink for patching is that GC stages patches and offers the benefits of its Patch Cache (available on the Deployments tab by clicking View Patch Cache). The Patch Cache keeps a record in the Repository of all downloaded patches, and allows you to apply a patch to multiple destinations in the same operation while only having to stage it once, which is particularly useful when patching multiple Agents. I am limiting this discussion to staging a one-off patch, not applying it, because most one-offs require shutting down all OMS hosts (prohibiting Console access) and/or stopping the Agent on the target host (preventing patch job submittal). See the patch Readme for directions on how to apply a staged one-off patch outside of Grid Control. Oracle Management Service Configuration Following are suggested OMS configuration changes to make. While none of these changes is strictly mandatory, I highly recommend all of them for tuning and customizing your OMS configuration. ■■ Check whether certain Grid Control log files are growing very large. ■■ Set up the oracle user environment on OMS hosts ■■ Modify the default Console timeout ■■ Configure the OMS for failover and load balancing (applies only to a RAC OMR database) ■■ Tune the OMS thread pool size 130 Oracle Enterprise Manager 10g Grid Control Implementation Guide UNIX Scripting Technique to Modify Configuration Files Some of the following post installation steps call for you to change OS configuration files. You can manually edit them with a text editor such as vi in UNIX or Notepad in Windows. Alternatively, you can employ the UNIX shell techniques below to append lines to a file or to make substitutions in a file. You can also use these techniques on Windows if running a UNIX shell on Windows, some of which are available as freeware. To append lines to a file called filename, use the following inline scripting technique: cat >> filename < EOF To make substitutions in a file, use the UNIX command, sed, in conjunction with the vi substitute command, s. For instance, to change all occurrences of SCOTT to TIGER in filename and to save these changes in newfilename, issue the following command (file entries changed are case sensitive): sed 's/SCOTT/TIGER/g' filename > newfilename The more files you need to change, the more time you will save using these techniques. As described in later chapters, you may need to make changes to certain configuration files, such as the Agent emd.properties file. If you are monitoring scores of target hosts, rather than manually changing each file, you can save a lot of time by writing an OS script to make the required changes and executing this script in one operation on all target hosts using one of two Grid Control eatures: “The Execute Host Command Feature” in Chapter 9, or the GC job system. If you need to bounce a component (such as an Agent) to make the change effective, you can write a script that changes the configuration file, then restarts the component. For an excellent example, see OracleMetalink Note 560905.1. ■■ Add an OMR alias to the OMS tnsnames.ora file ■■ Back up critical OMS files Reduce Grid Control Logging Now that you have patched Grid Control, it’s time to check that you are not hitting a particular Grid Control bug described below that causes an Oracle Notification Server (ONS) log file to grow very large within hours. In addition, because some OracleAS default logging levels on OMS hosts cause certain logs to grow large over time, we learn below how to restrict or disable logging and/or rotate and purge logs to reduce their footprint on the OMS filesystem. Without such measures, the OMS filesystem could eventually fill up, which would halt most GC operations, notably uploads of Agent metric alerts and data. Check for Abnormal ons.log Growth The Oracle Notification Server (ONS) running in the OMS home utilizes the ports defined in the $OMS_HOME/opmn/conf/ons.config file. The OMR Database listener, which is an ONS client, Chapter 4: Grid Control Post Installation 131 also uses the ONS ports identified in its own $ORACLE_HOME/opmn/conf/ons.config file. When Grid Control installs OracleAS and Oracle Database 10g on the same server, particularly when upgrading a Grid Control 10.1 installation to 10.2, the installer sometimes mistakenly configures identical ONS ports in both homes, which creates an operational conflict when both the OracleAS ONS and 10g RDBMS listener services are running. In these instances, ONS log files called $OMS_HOME/opmn/logs/ons.log.* or $ORACLE_HOME/opmn/logs/ons.log.* (where a timestamp is appended) begin growing exponentially large to gigabytes in size due to repeating messages of the following form6: Local connection 0,127.0.0.1,6100 missing form factor If you encounter these error messages, disable the OMR Database listener from subscribing to the Oracle Notification Server (ONS). To implement this solution, rename the OMR Database $ORACLE_HOME/opmn/conf/ons.config file so that the OMR Database listener cannot reference this file, then restart the OMR Database listener. Grid Control will not experience any problems when using this workaround. Reduce OMS Host Logging The Oracle Application Server (OracleAS) bundled with Grid Control contains the following subcomponents that generate their own log files: ■■ Oracle HTTP Server ■■ OC4J (applications that run in an OC4J instance within OracleAS): ■■ EM application ■■ home J2EE application ■■ Oracle Process Manager and Notification Server (OPMN) ■■ Web Cache Some of these log files, such as access logs and trace logs, are verbose and devoid of diagnostic information, but can grow to hundreds of megabytes or gigabytes in size over time and consume inordinate OMS filesystem space. You can reduce the amount of disk space these log files consume on the OMS filesystem by rotating and purging rotated log files, reducing logging levels, or disabling logging altogether. Some OracleAS subcomponents include a facility to rotate and purge log files. Other subcomponents can only rotate logs, but you can purge them easily enough at the operating system level. Table 4-1 below contains details of OracleAS subcomponent logging in Grid Control, including log file names, descriptions, configuration files controlling logging behavior, and logging strategy needed, if any. Follow the instructions below to reduce OMS host logging for the Oracle HTTP Server and OC4J subcomponents.7 Default OPMN and Web Cache logging should be sufficient, but I provide the logging directories and configuration files for these subcomponents in case you’re interested in controlling their logging as well. 6 7 Bug 4621067. Also see Note 284602.1 for details. The procedure here for reducing HTTP and OC4J logging is based on Note 339819.1. 132 Oracle Enterprise Manager 10g Grid Control Implementation Guide Subcomponent of OracleAS Log File Location Description of Log File Configuration File That Controls Logging Behavior Logging Strategy Oracle HTTP Server (OHS) $OMS_HOME/Apache/Apache/ logs/error_log.