Syslog And SNMP Monitoring Guide
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 102
| Download | |
| Open PDF In Browser | View PDF |
Luna SA Syslog and SNMP Monitoring Guide Luna Monitoring Syslog & SNMP Table of Contents Overview.....................................................................................................................................................................6 How to Use this Document.........................................................................................................................................6 Caveat Lector ..............................................................................................................................................................6 Applicability ................................................................................................................................................................6 Conventions & Style Used in this Document ..............................................................................................................6 Syslog Introduction .....................................................................................................................................................7 Structure of a syslog Message ....................................................................................................................................7 lunalogs ......................................................................................................................................................................9 hsm.log .................................................................................................................................................................... 10 Audit Logs ................................................................................................................................................................ 11 Interpreting Logs ..................................................................................................................................................... 11 Configuring syslog.................................................................................................................................................... 11 Open Source Components....................................................................................................................................... 12 IPMI ..................................................................................................................................................................... 12 Threshold Events ............................................................................................................................................. 16 Discrete Events ................................................................................................................................................ 18 CPU Sensor ...................................................................................................................................................... 18 VRD, PCH and Inlet Sensors ............................................................................................................................. 19 IPMI Implementation ...................................................................................................................................... 19 Significant Log Messages ................................................................................................................................. 19 Insignificant Log Messages .............................................................................................................................. 21 Unlikely Log Messages ..................................................................................................................................... 21 Debug Messages .............................................................................................................................................. 25 SMART ................................................................................................................................................................. 26 SMART Attributes ............................................................................................................................................ 27 Significant Expected Log Messages ................................................................................................................. 28 Unexpected Log Messages .............................................................................................................................. 31 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 2 of 102 Luna Monitoring Syslog & SNMP SNMP ................................................................................................................................................................... 45 Expected Log Messages ................................................................................................................................... 45 Unexpected Log Messages .............................................................................................................................. 46 NTP ...................................................................................................................................................................... 46 SSH ....................................................................................................................................................................... 46 Expected Log Messages ................................................................................................................................... 46 Unexpected Log Messages .............................................................................................................................. 47 Privileged Services ............................................................................................................................................... 48 Expected Log Messages ................................................................................................................................... 48 Unexpected Log Messages .............................................................................................................................. 49 Kernel................................................................................................................................................................... 50 Expected Log Messages ................................................................................................................................... 50 SafeNet Components............................................................................................................................................... 50 sysstat .................................................................................................................................................................. 51 Expected Log Messages ................................................................................................................................... 51 Unexpected Log Messages .............................................................................................................................. 52 HSM Callback Service .......................................................................................................................................... 53 OAMP................................................................................................................................................................... 53 Unexpected Log Messages .............................................................................................................................. 54 Network Trust Link Service .................................................................................................................................. 54 Datapath: Expected Log Messages ................................................................................................................. 54 Datapath: Unexpected Log Messages ............................................................................................................ 58 Luna SA Command Processor: Expected Log Messages ................................................................................. 64 Luna SA Command Processor: Unexpected Log Messages ............................................................................ 66 Host Trust Link ..................................................................................................................................................... 70 Expected Log Messages ................................................................................................................................... 70 Unexpected Log Messages .............................................................................................................................. 71 Network Bus Service ............................................................................................................................................ 74 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 3 of 102 Luna Monitoring Syslog & SNMP Remote Backup Service ....................................................................................................................................... 74 Luna Administrative Shell: State and Status ....................................................................................................... 74 Expected Log Messages ................................................................................................................................... 74 Unexpected Log Messages .............................................................................................................................. 75 Luna Shell: Command Logging ............................................................................................................................ 76 Expected Log Messages ................................................................................................................................... 76 Unexpected Log Messages .............................................................................................................................. 76 cron & crontab..................................................................................................................................................... 77 Expected Log Messages ................................................................................................................................... 77 Unexpected Log Messages .............................................................................................................................. 78 Certificate Monitoring Daemon .......................................................................................................................... 79 Expected Log Messages ................................................................................................................................... 79 Unexpected Log Messages .............................................................................................................................. 80 Luna SNMP Trap Agent ........................................................................................................................................ 82 Expected Log Messages ................................................................................................................................... 82 Unexpected Log Messages .............................................................................................................................. 82 Admin API ............................................................................................................................................................ 84 Expected Log Messages ................................................................................................................................... 84 Unexpected Log Messages .............................................................................................................................. 85 Other Appliance Logging ......................................................................................................................................... 86 hsm information show ........................................................................................................................................ 86 Client Logging .......................................................................................................................................................... 88 cklog..................................................................................................................................................................... 88 HA log .................................................................................................................................................................. 88 SNMP Introduction .................................................................................................................................................. 89 Traps ........................................................................................................................................................................ 89 Configuring and Enabling Traps on Luna Appliance ............................................................................................ 89 Configuring Trap Notification .............................................................................................................................. 90 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 4 of 102 Luna Monitoring Syslog & SNMP Install net-snmp ............................................................................................................................................... 90 Create a Trap Handler ..................................................................................................................................... 91 Configure snmptrapd.conf .............................................................................................................................. 91 Add the SafeNet MIBs ..................................................................................................................................... 92 Enable Trap Packets Through the Firewall ...................................................................................................... 92 Disable SELinux ................................................................................................................................................ 92 Start the SNMP Trap Daemon ......................................................................................................................... 93 Testing Trap Events on Luna Appliance ............................................................................................................... 93 Using the Examples ......................................................................................................................................... 94 Luna Appliance Trap Events ................................................................................................................................ 94 Appliance MIB Overview ..................................................................................................................................... 94 Sub-System Log Reference .................................................................................................................................. 95 Fan Failure ....................................................................................................................................................... 96 Power Supply Failure ....................................................................................................................................... 97 Motherboard Failure ....................................................................................................................................... 98 Disk Drive Failure ............................................................................................................................................. 99 NTLS Failure ..................................................................................................................................................... 99 Crypto Failure .................................................................................................................................................. 99 Clock Failure .................................................................................................................................................. 100 Caveats .............................................................................................................................................................. 100 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 5 of 102 Luna Monitoring Syslog & SNMP Overview Monitoring a Luna HSM is an operational requirement for mission critical systems. SafeNet makes monitoring available through two principle facilities: the system log (syslog) and simple network management protocol (SNMP). This document describes the syslog facility and provides details for what various messages mean, the probability of observing them and what action to take if observed. It goes on to describe the SNMP traps supported by Luna appliances. How to Use this Document SafeNet prepared this document for two primary purposes. Firstly, the information within should provide sufficient guidance for an administrator to review log messages and determine a course of action for significant events. Secondly, the contents of this document should provide sufficient information for a developer to create or configure a service to scan log messages for significant events and automate action based on the message. Caveat Lector SafeNet reserves the right to revise log messages with each new release. Consequently, do not hard code pattern matching strings to parse log files. If you intend to match on specific text, create a solution that enables you to easily configure pattern strings. Applicability SafeNet updated this document for the Luna SA 6.2 product release (HSM firmware 6.23.0). Much of the material also applies to all previous product releases (i.e., 5.0.x, 5.1.x, 5.2.x, 5.3.x, 5.4.x and 6.0.x). A companion document –Monitoring for Luna IS –describes log messages for SafeNet’s Luna IS 6.x product. Conventions & Style Used in this Document So far, one author has developed this guide. It is far from complete. Over time, others will participate to extend and update the contents. This section describes the conventions and style the author set for this document. This section is important to you, the reader, so that you understand the format herein. Equally important, this section is for maintainers of the document to ensure consistency for future revisions. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 6 of 102 Luna Monitoring Syslog & SNMP The following conventions apply throughout the document. • • • • • • • • • Use the same date and time in all examples unless the time is relevant to the example (e.g., a cron entry). For this version of the document, the date and time are: 2012 Feb 29 12:05:01. Use myLuna for the hostname of the Luna appliance. Use 192.168.0.xxx for example IP addresses. These values maintain consistency in the examples and ensures that no customer- or SafeNet-specific network configuration shows in an example. Use [1234] for the process identifier. Organize log messages into two distinct groups with the following headings: “Expected Log Messages” and “Unexpected Log Messages.” Use a table like this example to identify the facility keyword, software process and log file for each new component to be described. Use prose – not a bullets or tables – to describe log messages. Try to maintain the established introductory pre-amble, “This message indicates …” The document uses this structure so that parts can be extracted and provided electronically (e.g., via email) for customer assistance. Use greater-than-less-than signs to delimit variable text in log messages (e.g.,). Developers or editors of this document, please review “Appendix A – Guidance for Maintainers of this Document” for additional information. Syslog Introduction Syslog is a standard logging facility, standardized within the Syslog working group of the IETF. Software processes use an API to generate messages that the syslog facility writes to organized log files. If configured, syslog can also send messages to remote syslog servers. IMPORTANT: Luna appliances use rsyslog. This facility provides the same features as syslog with the addition of reliable transport using TCP. Unless relevant to the accuracy of a point being made, this document uses the term syslog rather than rsyslog. Structure of a syslog Message Figure 1 shows an example of a syslog message. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 7 of 102 Luna Monitoring Syslog & SNMP Figure 1: Example syslog Message 1. 2. 3. 4. 5. 6. This field is the date and time. This field is the system host name. This field is the facility keyword, explained below. This field is the log severity level, explained below. This field is the software process that generated the log message. This field is a process-specific log message. Table 1 summarizes the facility keywords applicable for the Luna appliance. Facility Keyword kern user daemon auth syslog authpriv cron local# Facility Description kernel messages user-level messages system daemons security/authorization messages messages generated internally by syslogd security/authorization messages clock daemon local use #, where # is 0 to 7 Table 1: syslog Facility Keywords Table 2 summarizes the log severity levels. Severity Keyword emerg/panic alert critical err/error warn/warn notice info debug Severity Description System is unusable Action must be taken immediately Critical condition Error condition Warning condition Normal but significant condition Informational message Debug-level message Table 2: syslog Severity Levels The primary log file is messages but the Luna appliance also creates two other significant log files: lunalogs and hsm.log. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 8 of 102 Luna Monitoring Syslog & SNMP lunalogs lunalogs log messages follow a similar format as standard syslog messages with some slight differences. Figure 2 shows an example segment of a lunalogs message. The format up to the second field is identical to that for a syslog message. Figure 2: Example lunalogs Message 1. The facility keyword for lunalogs varies. A table in relevant sections identifies the facility keyword for the component that writes log messages to lunalogs. 2. This field is the application string, itemized below. 3. This field is the process identifier, if available. 4. lunalogs has a subsidiary severity level, itemized below. 5. This field is the Luna-specific error code associated with the lunalog entry. 6. This field is the description, the format and contents determined by the application identifier of the lunalogs message. In most cases, the description is a concise statement of the issue that led to the log entry (e.g., oamp – “Cobra SQL service online.”). In other cases, the description comprises multiple fields of information, described below in Table 5. Table 3 summarizes the application identifiers available in a lunalogs message. Application Identifier oamp Recover NTLS lunash cluster Luna PED Client hsm_login certmonitord pam_swift sysstatd AdminAPI Table 3: lunalogs Application Identifiers Table 4 summarizes the subsidiary log severity levels of lunalogs. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 9 of 102 Luna Monitoring Syslog & SNMP Severity Keyword critical error warning audit info debug Table 4: lunalogs Severity Levels Table 5 shows the application-specific description for the more comprehensive lunalogs messages. Application NTLS lunash Description Field : / Example #1: Client opened session 18478 : HSM1:Part171 : 192.168.0.100/40847 Example #2: Received a command LUNA_DESTROY_OBJECT and object handle 20262 : 192.168.0.100/40847 : : / Example #1: Lush user login : monitor : 192.168.0.100/40847 Example #2: Command: log show : monitor : 192.168.0.100/40847 Table 5: Application-Specific Description hsm.log hsm.log log messages are similar to syslog messages. The following example shows an hsm.log message. 2012 Feb 29 12:05:01 myLuna local6 err oamp[1962]: ERR: RTC: tamper 2 signal The facility keyword is local6. hsm.log has only three severity levels: • • • crit err info. The application responsible for generating hsm.log messages is oamp. The description field is the severity level followed by a string read directly from the HSM. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 10 of 102 Luna Monitoring Syslog & SNMP One log message varies slightly from the above description. The following message marks the beginning of a new log section on power up of the HSM. 2012 Feb 29 12:05:01 myLuna local6 info oamp[1962]: ----- HSM Logging started Audit Logs See the “Overview - Security Audit Logging and the Audit Role” section of the Luna product documentation for a description of audit logs. Interpreting Logs No hard and fast rules exist for how to parse and interpret logs for significant events. For example, a “notice” severity from the IPMI daemon could be significant (e.g., PSU failed) or simply status information (e.g., reading sensors). The following bullets provide some guidance on how to parse log messages. • • • • • Scan for “critical” severity log entries. These logs represent significant events. Scan for “error” severity log entries. In most cases, these logs represent significant events. Scan for “notify” severity log entries from the ipmievd process and look for “Failure detected asserted”, “Lower Critical going low”, “Upper Critical going high”, “Lower Non-Recoverable going low” and “Upper Non-Recoverable going high.” Scan for “crit” severity logs entries for smartd. Look for "Temperature changed" to track internal appliance temperature measured at the hard drive. Look for excessive conditions with the string "reached critical limit" (e.g., temperature). Scan for “CRASH AND BURN” in the logs. An instance of this string indicates a programming or logic error. Configuring syslog See the “syslog Commands” reference section of the Luna product documentation for details on how to configure syslog messages in the Luna appliance. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 11 of 102 Luna Monitoring Syslog & SNMP Open Source Components SafeNet uses several open source components that leverage the syslog facility. Notable open source components are: • • • • • • IPMI SMART NTP SNMP SSH Privileged services. Sections that follow describe syslog messages that each of these open source components can generate. IPMI The Intelligent platform monitoring interface (IPMI) is a hardware-level specification for monitoring the operation of a computer system. In the case of the Luna appliance, IPMI monitors sensors within the appliance. Broad categories of these sensors are fans, power supplies and the system motherboard. Table 6 identifies: the specific sensors; their locations; the specific text string that represents each sensor; the thresholds for a sensor; threshold values; and the unit of measure for each threshold. Subsequent subsections of this document delve into each of the columns of the table. Sensor & Location Cooling fan; left-most of three fans, rear-most blade within fan unit. Sensor Text Fan1A Threshold Threshold Value Measure & Hysteresis LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . LNR deassert Positive 500 LC deassert Cooling fan; left-most of three fans, front-most blade within fan unit. Fan1B LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . LNR deassert Positive 500 LC deassert Cooling fan; center of three fans, rear-most blade within fan unit. Fan2A LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . LNR deassert Positive 500 LC deassert Cooling fan; center of three fans, front-most blade within fan unit. Fan2B Luna Syslog and SNMP Monitoring Guide LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 12 of 102 Luna Monitoring Syslog & SNMP Sensor & Location Sensor Text Threshold Threshold Value LNR deassert Measure & Hysteresis Positive 500 LC deassert Cooling fan; right-most of three fans, rear-most blade within fan unit. Fan3A LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . LNR deassert Positive 500 LC deassert Cooling fan; right-most of three fans, front-most blade within fan unit. Fan3B LNR assert 1000 Speed RPM LC assert 2000 Negative 500 . LNR deassert Positive 500 LC deassert CPU temperature measured at the geometric center of the CPU package; located approximately center of appliance, offset to righthand side. Voltage Regulator Down; located approximately 3cm toward front from CPU. CPU . UNR assert 89 Temperature Degrees C UC assert 72 Negative 3 UNR deassert Positive 3 UC deassert VRD . UNR assert 100 Temperature Degrees C UC assert 90 Negative 3 UNR deassert Positive 3 UC deassert Platform Controller Hub; located approximately center of appliance, 10cm from rear. PCH . UNR assert 100 Temperature Degrees C UC assert 90 Negative 3 UNR deassert Positive 3 UC deassert Located approximately center of appliance, 20 cm from front. Inlet . UNR assert 45 Temperature Degrees C UC assert 39 Negative 3 UNR deassert Positive 3 UC deassert Centered DIMM of six sockets; approximately center of chassis, left of CPU. CHA DIMM 0 . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert Luna appliances do not have memory installed for this device. “ns” readings result for this sensor. CHA DIMM 1 . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 13 of 102 Luna Monitoring Syslog & SNMP Sensor & Location Luna appliances do not have memory installed for this device. “ns” readings result for this sensor. Sensor Text CHA DIMM 2 Threshold . Threshold Value Measure & Hysteresis UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert Left-most DIMM, approximately center of chassis, right of power supplies. CHB DIMM 0 . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert Luna appliances do not have memory installed for this device. “ns” readings result for this sensor. CHB DIMM 1 . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert Luna appliances do not have memory installed for this device. “ns” readings result for this sensor. CHB DIMM 2 . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert More information on this sensor is forthcoming in a future revision RAM TMax . UNR assert 97 Temperature Degrees C UC assert 87 Negative 3 UNR deassert Positive 3 UC deassert CPU core voltage. CPU_VCORE UC assert 1.44 Voltage Volts LC assert 0.63 No hysteresis 2.796 Voltage Volts . UC deassert LC deassert LC assert Motherboard battery. VBAT . LC deassert 3V supply used by south bridge chip. 3VSB No hysteresis UC assert 3.492 Voltage Volts LC assert 3.092 No hysteresis UC assert 3.492 Voltage Volts LC assert 3.092 No hysteresis . UC deassert LC deassert +3.3 volt rail. 3VMain . UC deassert LC deassert Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 14 of 102 Luna Monitoring Syslog & SNMP Sensor & Location +5 volt rail. Sensor Text +5V Threshold Threshold Value Measure & Hysteresis UC assert 5.304 Voltage Volts LC assert 4.692 No hysteresis UC assert 12.740 Voltage Volts LC assert 11.284 No hysteresis UNR assert 13.392 Voltage Volts LNR assert 11.232 No hysteresis . UC deassert LC deassert +12 volt rail. +12V . UC deassert LC deassert +12 volt monitor of right power supply unit. PSU1_+12V_value. UNR deassert LNR deassert UNR assert Temperature of right power supply unit. 115 Temperature Degrees C UNR deassert PSU1 Temp_value. Negative 1 Positive 1 LNR assert Fan speed of right power supply unit. +12 volt monitor of left power supply unit. 300 LNR deassert Speed RPM Negative 100 PSU1 FAN_value . Positive 100 UNR assert 13.392 Voltage Volts LNR assert 11.232 No hysteresis 115 Temperature Degrees C PSU2_+12V_value. UNR deassert LNR deassert UNR assert Temperature of left power supply unit. PSU2 Temp_value. UNR deassert Positive 1 LNR assert Fan speed of left power supply unit. Right power supply status – presence detected Left power supply status – presence detected Negative 1 LNR deassert 300 Speed RPM Negative 100 PSU2 FAN_value . Positive 100 PSU1_Status . PSU2_Status . Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 15 of 102 Luna Monitoring Syslog & SNMP Sensor & Location Sensor Text Threshold Threshold Value Measure & Hysteresis Key to Threshold UNR UC LNR LC Upper Non-Recoverable Threshold Upper Critical Threshold Lower Non-Recoverable Threshold Lower Critical Threshold Table 6: Luna Appliance Sensors Threshold Events Here is an example sensor log. 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Fan sensor Fan3B going low (Reading 0 < Threshold 2000 RPM) . Lower Critical Most of the sensors monitored by IPMI on the Luna appliance generate threshold-based events. Threshold events occur when a sensor reading changes by an amount that crosses a configured boundary defined in the sensor data repository. THE IPMI specification defines six thresholds; the Luna appliance generates log messages on four of them. The four thresholds, each with a short mnemonic, are: 1. 2. 3. 4. Upper Non-Recoverable (UNR) Upper Critical (UC) Lower Non-Recoverable (LNC) Lower Critical (LC). Upper and lower critical thresholds are ‘fatal’ and mean that the hardware is exceeding above or below a specified rating respectively. Upper and lower non-recoverable thresholds are ‘potential damage’ and mean that the hardware is in jeopardy or damaged. A sensor value can change in one of two ways. It can go to an active state which IPMI calls asserted. The second state is inactive or de-asserted. IPMI also allows for a sensor to have a hysteresis. The importance of having a hysteresis is to reduce the chance of a “flood” of events as the sensor hovers around a threshold. For example, if a fan sensor has a going-low threshold of 1000 RPM and the reading goes slightly above and below this value repeated, IPMI will generate an event for every assertion and de-assertion as the reading crosses the 1000 RPM threshold. With a hysteresis of 100, for example, the event would assert at 1000 RPM but not de-assert until the reading was 1100 RPM or greater. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 16 of 102 Luna Monitoring Syslog & SNMP Combining the thresholds, hysteresis and state changes, IPMI has a means to create an event to describe what a sensor is doing. Figure 3 from the IPMI specification illustrates event assertion and desertion at different thresholds. Figure 3: Event Assertion/Desertion Thresholds To apply the meaning of thresholds, hysteresis and states, consider a real sensor on the Luna appliance: a cooling fan. A normal fan speed varies but generally ranges between 4000 RPM and 6000 RPM. The fan speed can fall below a threshold (e.g., when it fails) or rise above a threshold (e.g., when the ambient air temperature rises and more cooling is necessary). By way of an example, assume that someone impedes the propeller of a fan blade to slow it down but not stop it entirely. After ten seconds, our antagonist impedes the fan blade so that it is barely turning. This sequence of events results in two log messages. The first represents a lower critical assertion going low; the second, a lower non-recoverable assertion going low. 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Fan sensor Fan3B going low (Reading 1600 < Threshold 2000 RPM) . Lower Critical 2012 Feb 29 12:05:11 myLuna local4 notice ipmievd: Fan sensor Fan3B Non-recoverable going low (Reading 500 < Threshold 1000 RPM) . Lower Correlating the information in these two log messages with the data in third, fourth and fifth columns of Table 6, note that the first log message results because the fan speed has dropped below 2000 RPM to a reading of 1600 RPM ( ). The second message results because the fan speed has now crossed the non-recoverable threshold of 1000 RPM( ). After some period, with one less fan to cool the appliance, it begins to heat up, leading to the baseboard management controller to command the remaining fans to speed up. At this time, our antagonist Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 17 of 102 Luna Monitoring Syslog & SNMP discontinues impeding the fan blade. It immediately begins to spin and joins the other five at a higherthan-normal RPM. When the BMC detects the temperature back at a normal value, the BMC commands the fans to return to their normal speeds. The following log message records this event, the sensor deasserting the last threshold change. Note that several seconds have elapsed after the fan threshold deserted the LNR event and IPMI detected the change and read the sensor. Hence, the reading of 15900 represents a sensor value when the fans are at maximum revolutions . 2012 Feb 29 12:05:21 myLuna local4 notice ipmievd: Fan sensor Fan3B recoverable going low (Reading 15900 < Threshold 1000 RPM) . Lower Non- Upon examining log messages, you might also observe instances where two messages similar to the following examples occur in close proximity: 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Fan sensor Fan3B going low (Reading 0 < Threshold 2000 RPM) . Lower Critical 2012 Feb 29 12:05:51 myLuna local4 notice ipmievd: Fan sensor Fan3B recoverable going low (Reading 15900 < Threshold 1000 RPM) . Lower Non- The first message is the event of the fan stopping. The second message is the fan speed going from very fast back to a normal range after it has become freed to spin again and join the other five fans to cool the overheating appliance. This second log message represents the de-assert LNR event: by the time IPMI has scanned and read the sensors, the fan has asserted the LNR event and subsequently de-asserted it. IPMI only generates the log message for the more recent of the two events. Discrete Events The Luna appliance reports three discrete events. 1. OEM #0xa1. Because the IPMI specification does not define a power reset function, an OEM sensor serves this purpose. This OEM sensor links to a system reset function. The BMC records an event in the event log for a system reset: by pushing the power button; through a BMC reset; or via an operating system reset. 2. Power Supply PSU1_Status . This event reports the presence (asserted) or absence (de-asserted) of the right power supply. 3. Power Supply PSU2_Status . This event reports the presence (asserted) or absence (de-asserted) of the left power supply. CPU Sensor The CPU temperature reflects the use of the platform environment control interface (PECI) specification used for thermal management in the Intel Core 2 Duo microprocessor. PECI reports a negative value that expresses the difference between the current temperature and the thermal throttle point at which the CPU takes action to reduce speed or shut down to prevent damage. In other words, the PECI value is a relative and not an absolute temperature. The value reported by IPMI is calculated as the measured PECI value + an Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 18 of 102 Luna Monitoring Syslog & SNMP offset + a margin. As an example, if the PECI value is -89, the offset is 92 and the margin is 0, the IPMI reported temperature is 11 degrees C. This value is representative of a Luna appliance operating in a climate controlled data center. The important point is to monitor the CPU value in relation to the upper critical threshold of 72 degrees C and not as an absolute measure of the temperature of this sensor. VRD, PCH and Inlet Sensors The VRD, PCK and Inlet sensor values reported by IPMI reflect real-time temperature values. They differ in readings because of the heat generated at the different locations within the chassis (e.g., inlet near fans, PCH consumes considerable power). IPMI Implementation The principal entity responsible for reporting IPMI events within the Luna appliance is a daemon called ipmievd. This daemon listens for events from the baseboard management controller (BMC) sent to the system event log (SEL). ipmievd polls the contents of the SEL for new events. When it receives a new event, the daemon logs it to syslog. ipmievd is a standard Linux component, used unchanged by SafeNet. The daemon is based on the ipmitool utility and it shares the same IPMI support and session setup options: some of the potential log messages from ipmievd originate from the ipmitool utility. ipmievd can potentially generate hundreds of different log messages. Of these potential messages, some are sensor-specific. Other messages are unlikely but, because SafeNet uses the Linux component unchanged, are possible. The first subsection that follows describes the sensor events of most interest. The succeeding three subsections show insignificant messages, followed by the unlikely ones and debug messages. Significant Log Messages The log messages described in this subsection are the ones that report pertinent sensor events. The IPMI daemon reports on the sensor types shown in Table 7. Sensor Type Fan Temperature Voltage Power Supply OEM Table 7: Luna Appliance Sensor Types Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 19 of 102 Luna Monitoring Syslog & SNMP Full Sensor Record Threshold Reading Figure 4: Threshold Event Message A full sensor record threshold reading message consists of the fields shown in Figure 4. 1. This field is the sensor type, one of values from Table 7. 2. This field is the specific sensor from column 2 of Table 6. 3. This field is the description, formed from the threshold of column 3 of Table 6 combined with the direction of the threshold. 4. This field is the trigger reading as reported by the BMC for the sensor. 5. This field is the comparison that resulted in the event. 6. This field is the threshold value of column 4 of Table 6 for the applicable threshold. 7. This field is the unit of measure from column 5 of Table 6. Reproducing all possible threshold messages within this document is impractical. Rather, a more reasonable solution is to provide guidance for how to parse threshold messages. 1. Parse for ipmievd and Threshold in the log message. 2. Determine whether the event is an assertion or de-assertion. Use fields 4, 5 and 6 for this step. The event is an assertion if the logical evaluation of the statement formed from these three fields is true (e.g., (Reading 1600 < Threshold 2000 RPM). Otherwise, the event is a de-assertion (e.g., Reading 15900 < Threshold 1000 RPM). 3. Parse the remaining fields to obtain the sensor type, specific sensor affected, the trigger reading, the threshold, the threshold value and the unit of measure. With the information of the above list, you have the information available to create custom messages and/or take specific actions for threshold events. Compact Sensor Record Asserted/Deasserted 2012 Feb 29 12:05:01 myLuna detected Asserted local4 notice ipmievd: Power Supply sensor PSU2_Status . - Failure 2012 Feb 29 12:05:01 myLuna detected Deasserted local4 notice ipmievd: Power Supply sensor PSU2_Status . - Failure The first field after “ipmievd:” is the sensor type: see Table 7. The text after “sensor” is the specific sensor from column 2 of Table 6. The text after the dash is the description of the event with a specific indication of either Asserted or De-asserted at the end of the message. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 20 of 102 Luna Monitoring Syslog & SNMP You should review these messages within the context of any administrative or maintenance activities carried out on the Luna appliance. The severity and action depends upon what is reported. In the two examples above, the first log message indicates that PSU2 failed (e.g., A/C lost) while the second message indicates that PSU2 once again became operational. Non-Specific Sensor Record with Description 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Power Supply sensor - Transition to Power Off 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: OEM sensor - OEM Specific 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Power Unit sensor - Failure detected* * Reported by BMC firmware on older Luna appliances. These messages represent log entries for non-specific sensor records as per the IPMI specification. The first field after “ipmievd:” is the sensor type: see Table 7. The text after the dash is the description of the event. Like the compact sensor records, the context of any administrative or maintenance activities carried out on the Luna appliance as well as the description affects the action appropriate for the log message. Insignificant Log Messages The following log messages represent start up notices for ipmievd. Reading Sensors 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Reading sensors... This message indicates that the IPMI monitoring daemon successfully started just prior to setting up the event handler. Waiting For Events 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Waiting for events... This message indicates that the IPMI monitoring daemon successfully established communication with the IPMI device driver to relay events. Unlikely Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. No ipmievd Messages in Log File If you do not see any ipmievd messages in the messages log file, check to see if rsyslogd is running. Kernel Panic 2012 Feb 29 12:05:01 myLuna local4 alert Luna Syslog and SNMP Monitoring Guide ipmievd: Linux kernel panic: mod xyz fai 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 21 of 102 Luna Monitoring Syslog & SNMP A “panic” is an unrecoverable system error detected by the Linux kernel. A panic can result from unhandled processor exceptions such as references to invalid memory addresses. In the case of an IPMIreported panic, the likely cause is a hardware failure such as a failed RAM cell, errors in the arithmetic functions of the processor or an overheating/damaged processor. If you find this message in the log files, review other messages around it for an indication of what might have failed. Power up the appliance again and monitor for similar events. OEM Record 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: IPMI Event OEM Record 0xa1 SafeNet knows of no OEM records for the Luna appliance. Sensor Record without Description 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Power Supply sensor 0xa2 Sensor records with description should be all that the daemon reports. Note that the difference between a sensor record with a description versus one without is the dash after the word “sensor” in the message. See the subsection “Full Sensor Record Threshold Reading“ for a discussion of the records you should be most interested in. Full Sensor Record for Discrete and OEM Events 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Power Supply sensor PSU1_Status . 2012 Feb 29 12:05:01 myLuna status message local4 notice ipmievd: Power Supply sensor PSU1_Status . some A “full sensor record” is one formatted as “sensor” . This formatting is for discrete and OEM sensor events for which the Luna appliance reports none of these types. Compact Sensor Record With No Event Direction 2012 Feb 29 12:05:01 myLuna detected local4 notice ipmievd: Power Supply sensor PSU2_Status . - Failure This formatting is for compact sensor records with no direction reported for the threshold change. SafeNet is not aware of any events that report a message according to this format. Sensor Record With Only Sensor Number 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd: Fan sensor 16 SafeNet is not aware of any events that report a message according to this format. BMC Event Message Buffer Failure 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Get BMC Global Enables command failed 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Get BMC Global Enables command failed: 71 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Set BMC Global Enables command failed Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 22 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Set BMC Global Enables command failed: 23 Before the IPMI daemon can log messages, it must first open the BMC event message buffer. Under normal circumstances, this operation succeeds. Unable to Receive IPMI Message 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Unable to receive IPMI message The Luna appliance loads the IPMI device drivers before starting the IPMI daemon. This message means that the request to the device driver for an IPMI event message failed. No Event Data 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: No data in event All IPMI messages for the Luna appliance have well-formed event data. This message means that the IPMI daemon read an event from the IPMI message buffer but the event had no data associated with it. Not An Event 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Type 9 is not an event Only the asynchronous event receive type should be available in the BMC event message buffer. This message means that the IPMI daemon read an event from the IPMI message buffer that was not the expected asynchronous receive type. Unable to Read from IPMI Device 2012 Feb 29 12:05:01 myLuna local4 critical ipmievd: Unable to read from IPMI device This message means that the IPMI daemon failed to read from the event message buffer while polling for events. Get SEL Info Command Failed 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Get SEL Info command failed 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Get SEL Info command failed: 80 These messages indicate that the IPMI daemon was unable to get the system event log information for the storage network function. SEL Buffer Use 2012 Feb 29 12:05:01 myLuna clearing the SEL buffer local4 warning ipmievd: SEL buffer used at 85%, please consider This message means that the sensor event log buffer is filling faster than events can be processed by the IPMI daemon. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 23 of 102 Luna Monitoring Syslog & SNMP SEL Buffer Overflow 2012 Feb 29 12:05:01 myLuna local4 alert logged until the SEL buffer is cleared ipmievd: SEL buffer overflow, no SEL message can be This message means that the sensor event log buffer is full and no new messages can be queued. Unable to Retrieve SEL Data 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Unable to retrieve SEL data This message means that the IPMI daemon was unable to receive sensor event log data when it should have. SEL Overflow on Check for Waiting Events 2012 Feb 29 12:05:01 myLuna local4 notice 2012 Feb 29 12:05:01 myLuna local4 alert logged until the SEL buffer is cleared 2012 Feb 29 12:05:01 myLuna clearing the SEL buffer ipmievd: SEL overflow is cleared ipmievd: SEL buffer overflow, no new SEL message will be local4 warning ipmievd: SEL buffer is 85% full, please consider The IPMI daemon should process sensor event log messages in a timely fashion. These messages mean that the daemon is falling behind in doing so. PID File Already Exists 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: PID file '/var/run/ipmievd.pid0' already exists. 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Perhaps another instance is already running. These messages suggest: a failure of the operating system or file structure within the appliance; or a configuration error within a SafeNet-provided component. Failed to Open PID File 2012 Feb 29 12:05:01 myLuna local4 err for writing. Check file permission. ipmievd: Failed to open PID file '/var/run/ipmievd.pid0' This message suggests a failure of the operating system or file structure within the appliance. Unable to Open SDR File 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Unable to open SDR for reading This message suggests a failure in the IPMI subsystem. Malloc Failure 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: ipmitool: malloc failure This message indicates that the system is low on virtual memory while attempting to create a cache for the sensor data repository. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 24 of 102 Luna Monitoring Syslog & SNMP IPMI Event Setup Failed 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Could not enable event message buffer 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Could not enable event receiver 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Error setting up Event Interface open These messages indicate a failure of the ipmievd daemon to establish the interface via the IPMI device driver to relay events. Error Waiting For Events 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Error waiting for events! This message indicates a failure by the daemon to wait for events relayed via the IPMI device driver. Unable to Load Event Interface 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Unable to load event interface This message indicates a failure by the daemon to establish the IPMI event interface. Invalid Interface 2012 Feb 29 12:05:01 myLuna local4 err ipmievd: Invalid Interface for OpenIPMI Event Handler: This message suggests a corrupted disk system or a configuration error: the only potential value for is “open.” Debug Messages SafeNet does not enable logging of debug messages from the IPMI daemon. If in future SafeNet does enable this level of log messages, this subsection describes the possible messages you might find in the log files. BMC Event Message Buffer Opened 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: BMC Event Message Buffer enabled This message indicates that the IPMI daemon successfully opened the BMC event message buffer. BMC Event Message Debug 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: netfn:6 cmd:5 ccode:80 This message shows the network function code, command and completion code of the raw IPMI message. SEL Watch Debug 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL count is 3 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL freespace is 1502 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL Percent Used: 80% 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL Overflow: false Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 25 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL Next ID: 00da 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL lastid is 00d9 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: Current SEL count is 4 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: Current SEL lastid is 00ac 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL count is 0 (old=3), resetting lastid to 0 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL count lowered, new SEL lastid is 0112 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: SEL Read ID: 023f 2012 Feb 29 12:05:01 myLuna local4 debug ipmievd: New Events These messages show details of the sensor event log at various processing points within the daemon. Sensors Cached 2012 Feb 29 12:05:01 myLuna ipmievd: Getting 8 bytes from SDR at offset 64 2012 Feb 29 12:05:01 myLuna ipmievd: SDR reservation cancelled. Sleeping a bit and retrying... 2012 Feb 29 12:05:01 myLuna ipmievd: Sensors cached These messages indicate that the IPMI daemon successfully generated a fast lookup cache for the sensor data repository. IPMI Event Setup 2012 Feb 29 12:05:01 myLuna ipmievd: Enabling event message buffer 2012 Feb 29 12:05:01 myLuna ipmievd: Enabling event receiver 2012 Feb 29 12:05:01 myLuna ipmievd: Getting 8 bytes from SDR at offset 64 These messages log the process to set up the IPMI event interface. SMART Self-Monitoring, Analysis and Reporting Technology – SMART – is a technology built into modern hard drives to monitor the health of the drive and anticipate failures. The two classes of failures are: predictable and unpredictable. Examples of predicable failures are increased number of bad sectors and increased temperature from wear; they typically occur from slow processes. Unpredictable failures occur without warning and suddenly; for example, a hard drive dropped may report more than a normalized number of read errors. The hard drives within Luna appliances implements SMART. A background process called the smart control daemon – smartd – starts automatically at power up and is responsible for monitoring the hard drive via the SMART technology in it. The daemon records its findings via syslog. Facility Keyword daemon Luna Syslog and SNMP Monitoring Guide Software Process smartd[pid] Log File messages 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 26 of 102 Luna Monitoring Syslog & SNMP SMART Attributes From the Luna shell, you can query the SMART attributes (Luna SA/SP: status disk; Luna IS: system disk). Figure 5 shows the SMART attributes reported from the shell command. ID# 1 3 4 5 7 9 10 11 12 192 193 194 196 197 198 199 200 ATTRIBUTE_NAME Raw_Read_Error_Rate Spin_Up_Time Start_Stop_Count Reallocated_Sector_Ct Seek_Error_Rate Power_On_Hours Spin_Retry_Count Calibration_Retry_Count Power_Cycle_Count Power-Off_Retract_Count Load_Cycle_Count Temperature_Celsius Reallocated_Event_Count Current_Pending_Sector Offline_Uncorrectable UDMA_CRC_Error_Count Multi_Zone_Error_Rate FLAG 0x002f 0x0027 0x0032 0x0033 0x002e 0x0032 0x0032 0x0032 0x0032 0x0032 0x0032 0x0022 0x0032 0x0032 0x0030 0x0032 0x0008 VALUE 200 154 100 200 200 099 100 100 100 200 200 115 200 200 200 200 200 WORST 200 152 100 200 200 099 100 100 100 200 200 092 200 200 200 200 200 THRESH 051 021 000 140 000 000 000 000 000 000 000 000 000 000 000 000 000 TYPE Pre-fail Pre-fail Old_age Pre-fail Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age Old_age UPDATED Always Always Always Always Always Always Always Always Always Always Always Always Always Always Offline Always Offline WHEN_FAILED - RAW_VALUE 0 1300 601 0 0 1383 0 0 601 138 462 28 0 0 0 1 0 Figure 5: SMART Attributes The ID# and ATTRIBUTE NAME columns are the SMART attribute defined by the drive vendor. Note that vendors have generally standardized on the attributes and their meaning. The FLAG column indicates the main purpose of each attribute; correlating to the text in the TYPE and UPDATED columns. Only the last two bits are of any significance. Bit 0 identifies whether the attribute is an advisory (0) or a pre-fail notification (1). An advisory means that the disk has exceeded its intended design life while a pre-fail notification means that the disk is predicated to fail within 24 hours. Bit 1 indicates whether attribute monitoring occurs strictly offline (0) or during offline and normal operations (1). The RAW_VALUE, VALUE, WORST and THRESH columns report on the specific attribute. The drive controller reports the RAW_VALUE. VALUE is normalized where a higher value is better. As VALUE drops (i.e., becomes worse), it approaches a threshold. THRESH is the value at which SMART reports a ‘threshold exceeded’ event. WORST is the most extreme reading recorded for the attribute. A VALUE of 100 is an initial default value. Consider attribute #174 (Temperature) as an example to put these four values in context. Figure 5 shows the current temperature reading of the drive as 28 degrees Celsius. Normalized, this reading is 115. A value of 100 therefore is 43 degrees Celsius. The worst reading recorded is a normalized value of 092 which translates to 51 degrees Celsius. This attribute has no threshold as it represents an advisory. Assume that the smart control daemon generated the following log message: Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 27 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Attribute: 1 Raw_Read_Error_Rate changed from 200 to 92 Device: /dev/sda, SMART Prefailure The next query of the SMART attributes might report the following information for attribute #1: ID# ATTRIBUTE_NAME 1 Raw_Read_Error_Rate FLAG 0x002f VALUE WORST THRESH TYPE 092 092 051 Pre-fail UPDATED Always WHEN_FAILED RAW_VALUE 37767785 If the drive controller reads a value that, once normalized, is less than or equal to the THRESH value, the daemon generates a critical log message and a query of the SMART attributes reports a VALUE less than or equal to THRESH as the following example shows. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Attribute: 1 Raw_Read_Error_Rate changed from 92 to 51 ID# ATTRIBUTE_NAME 1 Raw_Read_Error_Rate FLAG 0x002f Device: /dev/sda, SMART Prefailure VALUE WORST THRESH TYPE 051 051 051 Pre-fail UPDATED Always WHEN_FAILED RAW_VALUE FAILING_NOW 4528445 The WHEN_FAILED column indicates that the attribute has reached the THRESH value. Significant Expected Log Messages Banner at Start Up 2012 Feb 29 12:05:01 myLuna daemon info Copyright (C) 2002-8 Bruce Allen smartd[1234]: smartd version 5.38 [i686-redhat-linux-gnu] 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Home page is http://smartmontools.sourceforge.net/#012 These two messages are the first log entries written by the smart control daemon. When SafeNet updates the smart control daemon to a more recent version, the following messages more closely resemble the startup banner. 2012 Feb 29 12:05:01 myLuna daemon info linux-2.6.18-164.el5] (local build) smartd[1234]: 1411:smartd 6.2 2013-07-26 r3841 [i686- 2012 Feb 29 12:05:01 myLuna daemon info Christian Franke, www.smartmontools.org smartd[1234]: Copyright (C) 2002-13, Bruce Allen, Read Configuration File 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Opened configuration file /etc/smartd.conf 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Configuration file /etc/smartd.conf parsed. These messages indicate that the smart control daemon successfully opened and read the contents of the configuration file that contains the directives for how the daemon is to operate. When SafeNet updates the daemon to a newer release, the location of the configuration file will change as represented in the follow examples. 2012 Feb 29 12:05:01 myLuna /usr/local/etc/smartd.conf daemon info 2012 Feb 29 12:05:01 myLuna daemon info /usr/local/etc/smartd.conf parsed. Luna Syslog and SNMP Monitoring Guide smartd[1234]: Opened configuration file smartd[1234]: Configuration file 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 28 of 102 Luna Monitoring Syslog & SNMP Monitoring Devices at Start Up 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, opened 2012 Feb 29 12:05:01 myLuna behind SAT layer. daemon info smartd[1234]: Device /dev/sda: using '-d sat' for ATA disk 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, opened 2012 Feb 29 12:05:01 myLuna database. daemon info smartd[1234]: Device: /dev/sda, not found in smartd 2012 Feb 29 12:05:01 myLuna Autosave. daemon info smartd[1234]: Device: /dev/sda, enabled SMART Attribute 2012 Feb 29 12:05:01 myLuna Offline Testing. daemon info smartd[1234]: Device: /dev/sda, enabled SMART Automatic 2012 Feb 29 12:05:01 myLuna to "monitor" list. daemon info smartd[1234]: Device: /dev/sda, is SMART capable. Adding 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Monitoring 0 ATA and 1 SCSI devices These messages indicate that the smart control daemon was successful in starting up and identifies the type and number of each device the daemon is monitoring. For Luna appliances, only one serial ATA device applies. The daemon opens the device to query its capabilities [the first message in the examples above]. The daemon uses SCSI/ATA Translation (SAT) to enable communication with the drive [the second message]. In SAT mode, the daemon reopens the device [third message] and looks for it in an internal database [fourth message] but does not find an entry for the device. The smart.conf file sets the directives for the daemon and these directives include attribute autosave (-S on) and automatic offline testing (-o on) [fifth and sixth messages]. Turning on attribute autosave instructs the disk drive to use non-volatile storage for attributes such as error counters, power-up hours and other useful data so that these values do not get reset to zero the next time the device is power-cycled. Turning on automatic offline testing instructs the drive to update certain S.M.A.R.T. attributes every four hours. The disk used in the Luna appliance is SMART-capable [seventh message] so the daemon monitors it [eighth message]. In preparing this document, SafeNet reviewed the source code of a newer version of the smart control daemon. SafeNet plans to update the daemon at some future date. The next set of messages show the messages this new daemon outputs at start up. 2012 Feb 29 12:05:01 myLuna to 'sat' daemon info smartd[1234]: Device: /dev/sda, type changed from 'scsi' 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda [SAT], opened 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda [SAT], WDC WD2500BHTZ04JCPV0, S/N:WD-WX11EC1TE568, WWN:5-0014ee-058a177e6, FW:04.06A00, 250 GB 2012 Feb 29 12:05:01 myLuna database. daemon info smartd[1234]: Device: /dev/sda [SAT], not found in smartd 2012 Feb 29 12:05:01 myLuna Attribute Autosave. daemon info smartd[1234]: Device: /dev/sda [SAT], enabled SMART 2012 Feb 29 12:05:01 myLuna Automatic Offline Testing. daemon info smartd[1234]: Device: /dev/sda [SAT], enabled SMART Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 29 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna Adding to "monitor" list. daemon info smartd[1234]: Device: /dev/sda [SAT], is SMART capable. 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Monitoring 1 ATA and 0 SCSI devices 2012 Feb 29 12:05:01 myLuna daemon info is 28 Celsius (Min/Max ??/28) smartd[1234]: Device: /dev/sda [SAT], initial Temperature The smart control daemon opens the device [third message] and successfully queries it for device-specific information [fourth message]. Similar messages follow in the output. The newer daemon reports the initial temperature reading from a sensor built into the disk [tenth message]. Fork Into Background Process 2012 Feb 29 12:05:01 myLuna New PID=23509. daemon info smartd[1234]: smartd has fork()ed into background mode. This message indicates that the startup sequence for the smart control daemon completed successfully, leading to the daemon to fork into a background mode process as exhibited if you query the process list. root 23509 1 0 11:04 ? 00:00:00 /usr/sbin/smartd -q never Self Tests 2012 Feb 29 12:05:01 myLuna Self-Test. daemon info smartd[1234]: Device: /dev/sda, starting scheduled Short 2012 Feb 29 12:05:01 myLuna Self-Test. daemon info smartd[1234]: Device: /dev/sda, starting scheduled Long These messages record that the smart-enabled drive ran a short and/or long self-test. The short self-test runs daily; the long self-test runs weekly, an hour after the short test. These tests usually run early in the morning. Attribute Change 2012 Feb 29 12:05:01 myLuna daemon info Seek_Error_Rate changed from 100 to 200 smartd[1234]: Device: /dev/sda, SMART Usage Attribute: 7 2012 Feb 29 12:05:01 myLuna daemon info Power_On_Hours changed from 100 to 99 smartd[1234]: Device: /dev/sda, SMART Usage Attribute: 9 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: 200 Multi_Zone_Error_Rate changed from 100 to 200 Device: /dev/sda, SMART Usage Attribute: These messages show changes in SMART attributes. The log severity of info means that the new values are still within acceptable ranges. Prefailure Change 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 100 to 200 Messages such as this example show changes in SMART pre-failure attributes. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 30 of 102 Luna Monitoring Syslog & SNMP Temperature Change 2012 Feb 29 12:05:01 myLuna daemon info Celsius to 37 Celsius (Min/Max 26/37) smartd[1234]: Device: /dev/sda, Temperature changed +5 This message is common in the log file and indicates a change in temperature measured at the hard disk sensor. The smart control daemon monitors the sensor once every 30 minutes and logs the temperature if it has changed from the last reading. Temperature Limit Reached 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: reached limit of 44 Celsius (Min/Max 31/49) Device: /dev/sda, Temperature 45 Celsius This message indicates that the temperature measured at the hard disk drive reached an upper limit for normally expected readings. Critical Temperature Change 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: reached critical limit of 48 Celsius (Min/Max 31/49) Device: /dev/sda, Temperature 49 Celsius This message indicates that the temperature measured at the hard disk drive exceeds the critical upper limit. Received Exit Signal 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: smartd received signal 15: Terminated 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: smartd received signal 3: Quit 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: smartd received signal <#>: These messages indicate that the smart control daemon received an exit signal. info severity messages are normal and expected; crit severity messages are not. In the latter case, the signal number is <#> and the text translation of this signal is . Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Configuration File not in Expected Location 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: No configuration file /usr/local/etc/smartd.conf found, scanning devices At start up, the smart control daemon expects to find the configuration file for its directives in the /usr/local/etc directory. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 31 of 102 Luna Monitoring Syslog & SNMP Cannot Open Configuration File 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: : Unable to open configuration file /usr/local/etc/smartd.conf This message indicates that although the smart control daemon located the configuration file, the daemon was unable to open it for the reason cited by in the log message. Configuration File Format Errors 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Error: line 14 of file /usr/local/etc/smartd.conf is more than MAXLINELEN=256 characters. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Error: continued line 44 (actual line 14) of file /usr/local/etc/smartd.conf is more than MAXCONTLINE=1023 characters. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 1 (drive DEVICESCAN): unknown Directive: /dev/sd4 2012 Feb 29 12:05:01 myLuna Directives. daemon crit smartd[1234]: Run smartd –D to print a list of valid 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 15 (drive /dev/sda): -s argument "(" is INVALID extended regular expression. Unmatched ( or \(. 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: File /usr/local/etc/smartd.conf line 15 (drive /dev/sda): ignoring previous Test Directive -s (S/../.././02) 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: File /usr/local/etc/smartd.conf line 15 (drive /dev/sda): warning, character 4 (x) looks odd in extended regular expression (S/x./.././02) 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: File /usr/local/etc/smartd.conf line 15 (drive /dev/sda): ignoring previous Address Directive -m john@email.com 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Configuration file /usr/local/etc/smartd.conf has fatal syntax errors. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Configuration file /usr/local/etc/smartd.conf parsed but has no entries (like /dev/hda) 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 16 (drive /dev/sda): Directive -M 'exec' argument: missing closing quote 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 16 (drive /dev/sda): Directive -M 'exec' argument must be followed by executable path. 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: File /usr/local/etc/smartd.conf line 16 (drive /dev/sda): ignoring previous mail Directive -M exec /usr/local/smtp 2012 Feb 29 12:05:01 myLuna daemon crit (drive /dev/sda): unknown Directive: -b smartd[1234]: File /usr/local/etc/smartd.conf line %18 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line %19 (drive /dev/sda): Missing argument to -W Directive 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line %20 (drive /dev/sda): Invalid argument to -M Directive: yearly 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Valid arguments to -n Directive are: never[,N][,q], sleep[,N][,q], standby[,N][,q], idle[,N][,q] * 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Drive: /dev/sda, implied '-a' Directive on line 21 of file /usr/local/etc/smartd.conf 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Drive: /dev/sda, -M Directive(s) on line 22 of file /usr/local/etc/smartd.conf need -m ADDRESS Directive Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 32 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Drive: /dev/sda, -m Directive on line 23 of file /usr/local/etc/smartd.conf needs -M exec Directive 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 35 (drive /dev/sda): Directive: -i takes integer argument from 1 to 255. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 36 (drive /dev/sda): Directive: -I has argument: on; needs integer from 1 to 255. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 37 (drive /dev/sda): Directive: -W takes 1-3 integer argument(s) from 0 to 255. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: File /usr/local/etc/smartd.conf line 38 (drive /dev/sda): Directive: -W has argument: off; needs 1-3 integer(s) from 0 to 255. These messages indicate format errors in the smartd.conf file. Depending upon the version of the smart control daemon, the configuration file might be /etc/smartd.conf or /usr/local/etc/smartd.conf. * This text is an example. See ‘man smartd.conf’ for a complete list and description of directives and arguments. Cannot Register Device 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Unable to register scsi device /dev/sda at line 17 of file /usr/local/etc/smartd.conf 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Unable to register scsi device /dev/sda These messages indicate that the smart control daemon was unable to register the device specified in the smartd.conf file. Force Scan For Devices 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Configuration file /usr/local/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices This message indicates that the configuration file specified that that the smart control daemon was to scan for devices rather than rely on entries in the file. Could Not Scan For Devices or No Devices To Scan 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Problem creating device name scan list 2012 Feb 29 12:05:01 myLuna found to scan daemon crit smartd[1234]: In the system's table of devices NO devices The first message indicates that the configuration file specified that that the smart control daemon was to scan for devices but that the scan failed. The second message indicates that daemon has no SMARTenabled devices to scan. Out of Memory 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Smartd: Out of memory This message indicates that the smart control daemon did not have sufficient memory on start up. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 33 of 102 Luna Monitoring Syslog & SNMP Exception 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Smartd: Exception: This message indicates that the smart control daemon encountered an exception on startup and exited. Received Hang Up or Interrupt Signal 2012 Feb 29 12:05:01 myLuna /etc/smartd.conf daemon info smartd[1234]: Signal HUP - rereading configuration file 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Signal INT - rereading configuration file /etc/smartd.conf ("SIGQUIT_KEYNAME" quits) These messages indicate that the smart control daemon was instructed via a signal to reread the configuration file normally read on start up. Continue on Error 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Reusing previous configuration This message indicates that while rereading the configuration file, the daemon encountered an error and reverted to the previously-read configuration settings instead. Unable to Monitor Devices 2012 Feb 29 12:05:01 myLuna daemon info Try debug (-d) option. Exiting... smartd[1234]: Unable to monitor any SMART enabled devices. This message indicates that the smart control daemon did not find any SMART-enabled devices to monitor. Mail-Related Messages 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Mail can't be enabled together with -capabilities. All mail will be suppressed. 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Sending a mail was suppressed. be send when capabilites [sic] are enabled 2012 Feb 29 12:05:01 myLuna cfg.mailwarn->emailfreq=0 daemon crit Mails can't smartd[1234]: internal error in MailWarning(): 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Contact smartmontoolssupport@lists.sourceforge.net; internal error in MailWarning(): which=-1, size=88 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Sending warning via to ... 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Executing test of to ... 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Warning via to : failed (fork or pipe failed, or no memory) 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Warning via to produced unexpected output (here truncated to 1024 bytes) to STDOUT/STDERR: 2012 Feb 29 12:05:01 myLuna remaining STDOUT/STDERR daemon crit Luna Syslog and SNMP Monitoring Guide smartd[1234]: Warning via to : flushed 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 34 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Warning via to : more than 1 MB STDOUT/STDERR flushed, breaking pipe 2012 Feb 29 12:05:01 myLuna failed daemon crit smartd[1234]: Warning via to : pclose(3) 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Warning via to : (32-bit/8-bit exit status: 255/133) perhaps caught signal 5 [Trace/breakpoint trap] failed 2012 Feb 29 12:05:01 myLuna daemon crit (32-bit/8-bit exit status: 255/133) smartd[1234]: Warning via to : failed 2012 Feb 29 12:05:01 myLuna smartd[1234]: Warning via to : successful daemon info 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Warning via to : exited because of uncaught signal 5 [Trace/breakpoint trap] 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Warning via to : process STOPPED because it caught signal 5 [Trace/breakpoint trap] 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, Read SMART Self-Test Log worked again, warning condition reset after 16 emails SafeNet does not compile the smart control daemon with libcap-ng enabled to send mail. The first two messages indicate that the daemon is compiled this way and that the settings in the configuration file are incompatible with libcap-ng settings. The latter messages indicate problems with how mail is configured to be sent and/or status of sending mail. One Check 2012 Feb 29 12:05:01 myLuna daemon info devices successfully [sic] checked once. smartd[1234]: Started with '-q onecheck' option. All smartd is exiting (exit status 0) The message indicates that the smartd.conf file contains a directive for the smart control daemon to start and terminate with only a check of any SMART-enabled devices. Ignore Device or Duplicate Device 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda[auto], ignored 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, duplicate, ignored The first message indicates that the smartd.conf file identifies a device but directs the smart control daemon to ignore the device for monitoring. The second message indicates that the device is specified more than once in the smartd.conf file. Could Not Detect Device, Unsupported Device, Not a ATA/SCSI Device or Device Cannot be Registered 2012 Feb 29 12:05:01 myLuna device type daemon info smartd[1234]: Device: /dev/sda, unable to autodetect 2012 Feb 29 12:05:01 myLuna 'auto' daemon info smartd[1234]: Device: /dev/sda, unsupported device type 2012 Feb 29 12:05:01 myLuna device daemon info smartd[1234]: Device: /dev/sda, neither ATA nor SCSI 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, not available Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 35 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna daemon crit Directive -d removable). Exiting. smartd[1234]: Unable to register device /dev/sda (no The first message indicates that the smart control daemon was unable to interrogate the device specified in the smartd.conf file. The second message indicates that the device is not SMART-enabled. The third message indicates that the device is not ATA or SCSI. The fourth message is a warning that the daemon could not detect the device but is continuing while the fifth message indicates that the daemon is stopping because it could not detect the device and the smartd.conf file contains a directive to do so under this circumstance. Could Not Open Device 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, open() failed: If debug is enabled or scanning is not, this message indicates that the smart control daemon was unable to open the device for monitoring: is the reason why. Could Not Fork Into Background Process 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: smartd unable to fork daemon process! 2012 Feb 29 12:05:01 myLuna up! daemon crit smartd[1234]: PID file /var/run/smartd.pid0 didn't show 2012 Feb 29 12:05:01 myLuna daemon crit /var/run/smartd.pid0 - exiting. smartd[1234]: unable to write PID file These messages indicate that the startup sequence for the smart control daemon was unable to complete successfully because the daemon could not properly fork a new process. Writing PID File 2012 Feb 29 12:05:01 myLuna PID 1729 daemon info smartd[1234]: file /var/run/smartd.pid0 written containing This message identifies the process identifier and process identifier file for the background smart control daemon process. Help for Configuration File Directives 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]:Configuration file (/usr/local/etc/smartd.conf) Directives (after device name): -d TYPE Set the device type: auto, ignore, removable ... -T TYPE Set the tolerance to one of: normal, permissive -o VAL Enable/disable automatic offline tests (on/off) -S VAL Enable/disable attribute autosave (on/off) -n MODE No check if: never, sleep[,N][,q], standby[,N][,q], idle[,N][,q] -H Monitor SMART Health Status, report if failed -s REG Do Self-Test at time(s) given by regular expression REG -l TYPE Monitor SMART log or self-test status: Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 36 of 102 Luna Monitoring Syslog & SNMP error, selftest, xerror, offlinests[,ns], selfteststs[,ns] -l scterc,R,W -e Set SCT Error Recovery Control Change device setting: aam,[N|off], apm,[N|off], lookahead,[on|off], security-freeze, standby,[N|off], wcache,[on|off] -f Monitor 'Usage' Attributes, report failures -m ADD Send email warning to address ADD -M TYPE Modify email warning behavior (see man page) -p Report changes in 'Prefailure' Attributes -u Report changes in 'Usage' Attributes -t Equivalent to -p and -u Directives -r ID Also report Raw values of Attribute ID with -p, -u or -t -R ID Track changes in Attribute ID Raw value with -p, -u or -t -i ID Ignore Attribute ID for -f Directive -I ID Ignore Attribute ID for -p, -u or -t Directive -C ID[+] Monitor [increases of] Current Pending Sectors in Attribute ID -U ID[+] Monitor [increases of] Offline Uncorrectable Sectors in Attribute ID -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit -v N,ST Modifies labeling of Attribute N (see man page) -P TYPE Drive-specific presets: use, ignore, show, showall -a Default: -H -f -t -l error -l selftest -l selfteststs -C 197 -U 198 -F TYPE Use firmware bug workaround: ... Attribute ID is a decimal integer 1 <= ID <= 255 Use ID = 0 to turn off -C and/or -U Directives Example: /dev/sda -a This text is the help information for configuring the smartd.conf file. Write State File 2012 Feb 29 12:05:01 myLuna /var/log/devstate daemon info smartd[1234]: Device: /dev/ada, state written to This message indicates that the smart control daemon wrote the state(s) of monitored device(s) to a file. Cannot Unlink PID File 2012 Feb 29 12:05:01 myLuna ( ). daemon crit smartd[1234]: Can't unlink PID file /var/run/smartd.pid0 This message indicates that the smart control daemon was unable to delete a process identifier file because of the error reported: . Close Failed 2012 Feb 29 12:05:01 myLuna failed daemon info Luna Syslog and SNMP Monitoring Guide smartd[1234]: Device: /dev/ada, , close() 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 37 of 102 Luna Monitoring Syslog & SNMP This message indicates that the smart control daemon was unable to close a monitored device because of the error reported: . Code Bug 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Please inform smartmontoolssupport@lists.sourceforge.net, including output of smartd -V. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: smartd is exiting (exit status 10) These messages indicate a coding error in the smart control daemon. ATA Error Log Read Failure 2012 Feb 29 12:05:01 myLuna Log failed daemon info smartd[1234]: Device: /dev/ada, Read Summary SMART Error 2012 Feb 29 12:05:01 myLuna daemon info Comprehensive SMART Error Log failed smartd[1234]: Device: /dev/ada, Read Extended 2012 Feb 29 12:05:01 myLuna Failed smartd[1234]: Device: /dev/ada, Read SMART Self Test Log daemon info These messages indicate a failure to read the error log(s) of ATA devices. ATA Device Scan Messages 2012 Feb 29 12:05:01 myLuna DEVICE Structure daemon info smartd[1234]: Device: /dev/sda, not ATA, no IDENTIFY 2012 Feb 29 12:05:01 myLuna daemon info device CD/DVD] not SMART capable smartd[1234]: Device: /dev/sda, packet devices [this 2012 Feb 29 12:05:01 myLuna daemon info searched (Directive: -P ignore). smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, found in smartd database. 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Device: /dev/sda, WARNING: i 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, presets are: ... 2012 Feb 29 12:05:01 myLuna Device: /dev/ada, smartd database not daemon info smartd[1234]: Device: /dev/ada, lacks SMART capability 2012 Feb 29 12:05:01 myLuna daemon info 82-83 don't specify if SMART capable. smartd[1234]: Device: /dev/ada, ATA IDENTIFY DEVICE words 2012 Feb 29 12:05:01 myLuna daemon info permissive' Directive given. smartd[1234]: Device: /dev/ada, proceeding since '-T 2012 Feb 29 12:05:01 myLuna T permissive' Directive. daemon info smartd[1234]: Device: /dev/ada, to proceed anyway, use '- 2012 Feb 29 12:05:01 myLuna capability daemon info smartd[1234]: Device: /dev/ada, could not enable SMART 2012 Feb 29 12:05:01 myLuna Attribute Autosave. daemon info smartd[1234]: Device: /dev/ada, could not disable SMART 2012 Feb 29 12:05:01 myLuna Autosave. daemon info smartd[1234]: Device: /dev/ada, disabled SMART Attribute 2012 Feb 29 12:05:01 myLuna Attribute Autosave. daemon info smartd[1234]: Device: /dev/ada, could not enable SMART Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 38 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna Health Status check daemon info smartd[1234]: Device: /dev/ada, not capable of SMART 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, Read SMART Values failed 2012 Feb 29 12:05:01 myLuna daemon info failed, ignoring -f Directive smartd[1234]: Device: /dev/ada, Read SMART Thresholds 2012 Feb 29 12:05:01 myLuna daemon info Temperature, ignoring -W 3,44,48 smartd[1234]: Device: /dev/ada, can't monitor 2012 Feb 29 12:05:01 myLuna ignoring -r 279! daemon info smartd[1234]: Device: /dev/ada, no Attribute 279, 2012 Feb 29 12:05:01 myLuna daemon info Attributes, ignoring -r 196! smartd[1234]: Device: /dev/ada, not monitoring Prefailure 2012 Feb 29 12:05:01 myLuna Automatic Offline Testing. daemon info smartd[1234]: Device: /dev/ada, could not disable SMART 2012 Feb 29 12:05:01 myLuna Testing unsupported... daemon info smartd[1234]: Device: /dev/ada, SMART Automatic Offline 2012 Feb 29 12:05:01 myLuna Offline Testing failed. daemon info smartd[1234]: Device: /dev/ada, enable SMART Automatic 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: ignoring -l selftest (override with -T permissive) Device: /dev/ada, no SMART Self-test Log, 2012 Feb 29 12:05:01 myLuna ignoring -l selftest Device: /dev/ada, no SMART Self-test Log, daemon info smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: ignoring -l error (override with -T permissive) Device: /dev/ada, no SMART Error Log, 2012 Feb 29 12:05:01 myLuna ignoring -l error Device: /dev/ada, no SMART Error Log, daemon info smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, no Extended Comprehensive SMART Error Log, ignoring -l xerror (override with -T permissive) 2012 Feb 29 12:05:01 myLuna daemon info SMART Error Log, ignoring -l xerror smartd[1234]: Device: /dev/ada, no Extended Comprehensive 2012 Feb 29 12:05:01 myLuna daemon info different error counts: 12 != 143 smartd[1234]: Device: /dev/ada, SMART Error Logs report 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, no SMART Offline Data Collection capability, ignoring -l offlinests (override with -T permissive) 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, no SMART Self-test capability, ignoring -l selfteststs (override with -T permissive) 2012 Feb 29 12:05:01 myLuna daemon crit support, ignoring -n Directive smartd[1234]: Device: /dev/ada, no ATA CHECK POWER STATUS 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: returned 17, not ATA compliant, ignoring -n Directive Device: /dev/ada, CHECK POWER STATUS 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, ATA settings applied: 2012 Feb 29 12:05:01 myLuna daemon info Control support, ignoring -l scterc smartd[1234]: Device: /dev/ada, no SCT Error Recovery 2012 Feb 29 12:05:01 myLuna Control failed daemon info smartd[1234]: Device: /dev/ada, set of SCT Error Recovery 2012 Feb 29 12:05:01 myLuna daemon info Control set to: Read: 7, Write: 7 smartd[1234]: Device: /dev/ada, SCT Error Recovery Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 39 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, state read from /var/lib/smartmontools/smartd.ST3500620AS-5QM2644Q.ata.state These messages indicate the status or a failure of a scan of an ATA device. The Luna appliance does not use ATA devices. SCSI Device Scan Messages 2012 Feb 29 12:05:01 myLuna INQUIRY failed; skip device daemon info smartd[1234]: Device: /dev/sda, Both 36 and 64 byte 2012 Feb 29 12:05:01 myLuna than 36 bytes; skip device daemon info smartd[1234]: Device: /dev/sda, INQUIRY response less 2012 Feb 29 12:05:01 myLuna [PDT=0x7], skip daemon info smartd[1234]: Device: /dev/sda, not a disk like device 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, WDC WD2500JS-60NCB1 2012 Feb 29 12:05:01 myLuna down); skip device daemon info smartd[1234]: Device: /dev/sda, NOT READY (e.g. spun 2012 Feb 29 12:05:01 myLuna device daemon info smartd[1234]: Device: /dev/sda, NO MEDIUM present; skip 2012 Feb 29 12:05:01 myLuna READY; skip device daemon info smartd[1234]: Device: /dev/sda, BECOMING (but not yet) 2012 Feb 29 12:05:01 myLuna [err=-22] daemon crit smartd[1234]: Device: /dev/sda, failed Test Unit Ready 2012 Feb 29 12:05:01 myLuna page, err=-5, skip device daemon info smartd[1234]: Device: /dev/sda, Bad IEC (SMART) mode 2012 Feb 29 12:05:01 myLuna skip device daemon info smartd[1234]: Device: /dev/sda, IE (SMART) not enabled, Try 'smartctl -s on /dev/sda' to turn on SMART features 2012 Feb 29 12:05:01 myLuna read SMART values daemon info smartd[1234]: Device: /dev/sda, unexpectedly failed to 2012 Feb 29 12:05:01 myLuna daemon info Temperature, ignoring -W 3,44,48 smartd[1234]: Device: /dev/sda, can't monitor 2012 Feb 29 12:05:01 myLuna Self-Test Log. daemon info smartd[1234]: Device: /dev/sda, does not support SMART 2012 Feb 29 12:05:01 myLuna autosave (set GLTSD bit). daemon info smartd[1234]: Device: /dev/sda, could not disable 2012 Feb 29 12:05:01 myLuna GLTSD bit). daemon info smartd[1234]: Device: /dev/sda, disabled autosave (set 2012 Feb 29 12:05:01 myLuna (clear GLTSD bit). daemon info smartd[1234]: Device: /dev/sda, could not enable autosave 2012 Feb 29 12:05:01 myLuna GLTSD bit). daemon info smartd[1234]: Device: /dev/sda, enabled autosave (cleared 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, state read from /var/lib/smartmontools/smartd.ST3500620AS-5QM2644Q.ata.state These messages indicate the status or a failure of a scan of a SCSI or SCSI-like device. The Luna appliance uses a serial ATA device which SMART considers SCSI-like. However, the messages in this section are unexpected in the logs. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 40 of 102 Luna Monitoring Syslog & SNMP Log Self-Test Progress & Errors 2012 Feb 29 12:05:01 myLuna completed without error daemon info smartd[1234]: Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna aborted by the host daemon info smartd[1234]: Device: /dev/sda, previous self-test was 2012 Feb 29 12:05:01 myLuna daemon info interrupted by the host with a reset smartd[1234]: Device: /dev/sda, previous self-test was 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: not complete due to a fatal or unknown error Device: /dev/sda, previous self-test could 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: completed with error (unknown test element) Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: completed with error (electrical test element) Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: completed with error (servo/seek test element) Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna daemon crit completed with error (read test element) smartd[1234]: Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna daemon crit completed with error (handling damage?) smartd[1234]: Device: /dev/sda, previous self-test 2012 Feb 29 12:05:01 myLuna 80% remaining daemon info smartd[1234]: Device: /dev/sda, self-test in progress, 2012 Feb 29 12:05:01 myLuna 0x1f daemon info smartd[1234]: Device: /dev/sda, unknown self-test status 2012 Feb 29 12:05:01 myLuna increased from 12 to 13 daemon crit smartd[1234]: Device: /dev/sda, Self-Test Log error count 2012 Feb 29 12:05:01 myLuna at hour timestamp 7202 daemon crit smartd[1234]: Device: /dev/sda, new Self-Test Log error 2012 Feb 29 12:05:01 myLuna decreased from 27 to 26 daemon info smartd[1234]: Device: /dev/sda, Self-Test Log error count 2012 Feb 29 12:05:01 myLuna Tests daemon crit smartd[1234]: Device: /dev/sda, does not support Self- 2012 Feb 29 12:05:01 myLuna already in progress daemon info smartd[1234]: Device: /dev/sda, skip since Self-Test 2012 Feb 29 12:05:01 myLuna Test daemon crit smartd[1234]: Device: /dev/sda, not capable of L Self- 2012 Feb 29 12:05:01 myLuna Test daemon crit smartd[1234]: Device: /dev/sda, not capable of Long Self- 2012 Feb 29 12:05:01 myLuna failed (err: -1) daemon crit smartd[1234]: Device: /dev/sda, execute Long Self-Test 2012 Feb 29 12:05:01 myLuna Self-Testing. daemon crit smartd[1234]: Device: /dev/ada, not capable of Offline or 2012 Feb 29 12:05:01 myLuna Self-Testing. daemon crit smartd[1234]: Device: /dev/ada, not capable of Offline or 2012 Feb 29 12:05:01 myLuna Immediate Test daemon crit smartd[1234]: Device: /dev/ada, not capable of Offline 2012 Feb 29 12:05:01 myLuna Self-Test daemon crit smartd[1234]: Device: /dev/ada, not capable of Conveyance Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 41 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna Self-Test daemon crit smartd[1234]: Device: /dev/ada, not capable of Short 2012 Feb 29 12:05:01 myLuna Test daemon crit smartd[1234]: Device: /dev/ada, not capable of Long Self- 2012 Feb 29 12:05:01 myLuna Self-Test daemon crit smartd[1234]: Device: /dev/ada, not capable of Selective 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, will not skip scheduled Offline Immediate Test despite unclear Self-Test byte (SAMSUNG Firmware bug). 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Immediate Test; 20% remaining of current Self-Test. Device: /dev/ada, skip scheduled Offline 2012 Feb 29 12:05:01 myLuna Test failed Device: /dev/ada, prepare Offline Immediate daemon crit smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: 1256500 - 1460000 2500000 sectors, 50% - 58% of disk). Device: /dev/ada, next test span at LBA 2012 Feb 29 12:05:01 myLuna Test failed. Device: /dev/ada, execute Offline Immediate daemon crit smartd[1234]: These messages indicate the status of an in-progress self-test, a failure to log self-tests or a self-test error. Test Schedule 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: run at Wed Feb 29 02:00:00 2012 GMT, starting now. Device: /dev/sda, old test of type S not 2012 Feb 29 12:05:01 myLuna each type per device): daemon info smartd[1234]: Next scheduled self tests (at most 5 of 2012 Feb 29 12:05:01 myLuna daemon info at Wed Feb 29 02:00:00 2012 GMT smartd[1234]: Device: /dev/sda, will do test 1 of type S 2012 Feb 29 12:05:01 myLuna Feb 29 02:00:00 2012 GMT]: daemon info smartd[1234]: Totals [Wed Feb 29 12:05:00 2012 GMT - Wed 2012 Feb 29 12:05:01 myLuna S daemon info smartd[1234]: Device: /dev/sda, will do 050 tests of type These messages indicate status of scheduled tests. Check Pending Sector Count Identifier 2012 Feb 29 12:05:01 myLuna daemon info Pending Sector count - no Attribute 197 smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Sector count - bogus Attribute 197 value 3442 (0xd72) Device: /dev/sda, can't monitor Current Device: /dev/sda, ignoring Current Pending These messages indicate an error on the sector count identifier for a SMART attribute. Critical Attribute Change 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Seek_Error_Rate changed from 10000 to 20000 Device: /dev/sda, SMART Usage Attribute: 7 Messages such as this example, show critical changes in SMART attributes, possibly indicating impending disk failure. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 42 of 102 Luna Monitoring Syslog & SNMP Critical Prefailure Change 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: Attribute: 1 Raw_Read_Error_Rate changed from 92 to 51 Device: /dev/sda, SMART Prefailure Messages such as this example, show critical changes in SMART pre-failure attributes, possibly indicating impending disk failure Sector Count Change 2012 Feb 29 12:05:01 myLuna daemon crit unreadable (pending) sectors smartd[1234]: Device: /dev/sda, 1237505 Currently 2012 Feb 29 12:05:01 myLuna (pending) sectors daemon crit smartd[1234]: Device: /dev/sda, 1237505 Total unreadable 2012 Feb 29 12:05:01 myLuna uncorrectable sectors daemon crit smartd[1234]: Device: /dev/sda, 1237505 Offline 2012 Feb 29 12:05:01 myLuna uncorrectable sectors daemon crit smartd[1234]: Device: /dev/sda, 1237505 Total offline These messages indicate sector attribute changes. Any one of these messages may include additional information at the end of the message to indicate the change from the last measure, [e.g., (changed 57500)]. Temperature Change 2012 Feb 29 12:05:01 myLuna Temperature daemon info smartd[1234]: Device: /dev/sda, failed to read 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: [trip Temperature is 65 Celsius] The first message indicates that the smart control daemon was unable to obtain a measure of the temperature attribute. The second message indicates the temperature at which the drive shuts down if it measures this value. Attribute Error 2012 Feb 29 12:05:01 myLuna daemon crit Attribute: 185 Head Stability. smartd[1234]: Device: /dev/sda, Failed SMART usage 2012 Feb 29 12:05:01 myLuna daemon info different ID numbers: 185 = 200 smartd[1234]: Device: /dev/sda, same Attribute has These messages indicate errors by the smart control daemon to interpret SMART attributes. Check ATA Device Health 2012 Feb 29 12:05:01 myLuna device or address daemon info smartd[1234]: Device: /dev/ada, open() failed: No such 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/ada, opened ATA device 2012 Feb 29 12:05:01 myLuna up disk (0x00 -> 0xff) daemon info smartd[1234]: Device: /dev/ada, CHECK POWER STATUS spins 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: returned 1, not ATA compliant, ignoring -n Directive Luna Syslog and SNMP Monitoring Guide Device: /dev/ada, CHECK POWER STATUS 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 43 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna suspending checks daemon info smartd[1234]: Device: /dev/ada, is in STANDBY mode, 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: reached limit of skipped checks (15 checks skipped) Device: /dev/ada, IDLE mode ignored due to 2012 Feb 29 12:05:01 myLuna daemon info mode, resuming checks (1 check skipped) smartd[1234]: Device: /dev/ada, is back in ACTIVE or IDLE 2012 Feb 29 12:05:01 myLuna self-check daemon crit smartd[1234]: Device: /dev/ada, not capable of SMART 2012 Feb 29 12:05:01 myLuna BACK UP DATA NOW! daemon crit smartd[1234]: Device: /dev/ada, FAILED SMART self-check. 2012 Feb 29 12:05:01 myLuna Attribute Data daemon crit smartd[1234]: Device: /dev/ada, failed to read SMART 2012 Feb 29 12:05:01 myLuna from 15 to 16 daemon crit smartd[1234]: Device: /dev/ada, ATA error count increased These messages provide health status of ATA device attributes. Check SCSI Device Health 2012 Feb 29 12:05:01 myLuna device or address daemon info smartd[1234]: Device: /dev/sda, open() failed: No such 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, opened SCSI device 2012 Feb 29 12:05:01 myLuna values daemon info smartd[1234]: Device: /dev/sda, failed to read SMART 2012 Feb 29 12:05:01 myLuna string> daemon crit smartd[1234]: Device: /dev/sda, SMART Failure: smartd[1234]: Device: /dev/sda, System auto standby 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, Disable auto standby not supported, ignoring ',ns' from -l offlinests,ns and -l selfteststs,ns 2012 Feb 29 12:05:01 myLuna daemon crit smartd[1234]: system auto standby enable failed: Device: /dev/sda, Self-test(s) completed, 2012 Feb 29 12:05:01 myLuna system auto standby enabled Device: /dev/sda, Self-test(s) completed, daemon info smartd[1234]: 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: system auto standby disable rejected: Device: /dev/sda, Self-test(s) in progress, 2012 Feb 29 12:05:01 myLuna daemon info system auto standby disabled Device: /dev/sda, Self-test(s) in progress, 1 smartd[1234]: If debug mode enabled. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 44 of 102 Luna Monitoring Syslog & SNMP These messages indicate failures of the auto standby feature of the hard disk. Sleep and Wakeup 2012 Feb 29 12:05:01 myLuna Resetting next wakeup time. daemon crit smartd[1234]: System clock time adjusted to the past. 2012 Feb 29 12:05:01 myLuna daemon info assuming wakeup from standby mode. smartd[1234]: Sleep time was 10 seconds too long, 2012 Feb 29 12:05:01 myLuna than in 30 seconds. smartd[1234]: Signal USR1 - checking devices now rather daemon info These messages indicate failures in the mechanism to sleep and wake up the smart control daemon at prescribed intervals to query and report on SMART attributes. Offline Log Collection Status 2012 Feb 29 12:05:01 myLuna was never started (auto:on) daemon info smartd[1234]: Device: /dev/sda, offline data collection 2012 Feb 29 12:05:01 myLuna daemon info was completed without error (auto:on) smartd[1234]: Device: /dev/sda, offline data collection 2012 Feb 29 12:05:01 myLuna is in progress (auto:on) smartd[1234]: Device: /dev/sda, offline data collection daemon info 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, offline data collection was suspended by an interrupting command from host (auto:on) 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: Device: /dev/sda, offline data collection was aborted by an interrupting command from host (auto:on) 2012 Feb 29 12:05:01 myLuna daemon info smartd[1234]: was aborted by the device with a fatal error (auto:on) Device: /dev/sda, offline data collection 2012 Feb 29 12:05:01 myLuna collection status 0x78 Device: /dev/sda, unknown offline data daemon info smartd[1234]: These messages indicate failures to perform offline data collection. Depending upon the status, the smart control daemon logs these messages as either crit or info; the examples above show info only. SNMP Luna appliances support monitoring via the simple network management protocol. A daemon, snmp, runs in the background on the appliance for this purpose. Facility Keyword user Software Process root Log File messages Expected Log Messages This revision of Luna Monitoring identifies only one expected log message. Starting Daemon 2012 Feb 29 12:05:01 myLuna user notice Luna Syslog and SNMP Monitoring Guide root: Starting snmpd 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 45 of 102 Luna Monitoring Syslog & SNMP This message indicates that the SNMP daemon is being started. Unexpected Log Messages A future revision of Luna Monitoring will describe unexpected log messages. NTP Network Time Protocol is an optional service of Luna appliances to synchronize the system clock to a coordinated universal time. A future revision of Luna Monitoring will describe NTP log messages. SSH Secure Shell or SSH is the process that offers secure, trusted connection to a restricted shell on the Luna appliance for administrative purposes. Facility Keyword authpriv Software Process sshd[pid] Log File secure This version of Luna Monitoring shows some examples of log messages you might find for SSH but is not a comprehensive account of all possible messages. Expected Log Messages 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Server listening on 0.0.0.0 port 22. 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Received signal 15; terminating. 2012 Feb 29 12:05:01 myLuna authpriv info 192.168.10.100 port 51286 ssh2 ssdh[1234]: Accepted password for from 2012 Feb 29 12:05:01 myLuna disconnected by user authpriv info ssdh[1234]: Received disconnect from 192.168.10.100: 11: 2012 Feb 29 12:05:01 myLuna 192.168.0.100 authpriv info ssdh[1234]: Did not receive identification string from 2012 Feb 29 12:05:01 myLuna authpriv info The user disconnected the application ssdh[1234]: Received disconnect from 192.168.0.100: 11: 2012 Feb 29 12:05:01 myLuna authpriv info from 192.168.0.100 port 2299 ssh2 ssdh[1234]: Accepted publickey for 2012 Feb 29 12:05:01 myLuna for user admin by (uid=0) authpriv info ssdh[1234]: pam_unix(ssdh[1234]:session): session opened 2012 Feb 29 12:05:01 myLuna for user admin authpriv info ssdh[1234]: pam_unix(ssdh[1234]:session): session closed 2012 Feb 29 12:05:01 myLuna authpriv info disconnected by server request ssdh[1234]: Received disconnect from 192.168.0.100: 2: 2012 Feb 29 12:05:01 myLuna ssdh[1234]: Connection closed by 192.168.0.100 [preauth] authpriv info These messages indicate normal SSH activity. is “admin”, “operator”, “monitor”, “audit” or a customer-defined name. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 46 of 102 Luna Monitoring Syslog & SNMP Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. 2012 Feb 29 12:05:01 myLuna Address already in use. authpriv err ssdh[1234]: error: Bind to port 22 on 0.0.0.0 failed: 2012 Feb 29 12:05:01 myLuna authpriv crit ssdh[1234]: fatal: Cannot bind any address. 2012 Feb 29 12:05:01 myLuna reset by peer [preauth] authpriv crit ssdh[1234]: fatal: Read from socket failed: Connection 2012 Feb 29 12:05:01 myLuna authpriv info failures for [preauth] ssdh[1234]: Disconnecting: Too many authentication 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Invalid user from 192.168.0.100 2012 Feb 29 12:05:01 myLuna name> [preauth] authpriv info ssdh[1234]: input_userauth_request: invalid user 2012 Feb 29 12:05:01 myLuna authpriv info because not listed in AllowUsers ssdh[1234]: User root from 192.168.0.100 not allowed 2012 Feb 29 12:05:01 myLuna from 10.168.64.4 authpriv info ssdh[1234]: Bad protocol version identification 'id' 2012 Feb 29 12:05:01 myLuna service "ssdh[1234]" authpriv warn ssdh[1234]: Deprecated pam_stack module called from 2012 Feb 29 12:05:01 myLuna authpriv notice ssdh[1234]: PAM 4 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.101.18.32 user=admin 2012 Feb 29 12:05:01 myLuna retries; 5 > 3 authpriv alert ssdh[1234]: PAM service(ssdh[1234]) ignoring max 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Received request to connect to host 127.0.0.1 port 3306, but the request was denied. 2012 Feb 29 12:05:01 myLuna authpriv err ssdh[1234]: error: Received disconnect from 192.168.0.100: 3: com.jcraft.jsch.JSchException: Auth fail [preauth] 2012 Feb 29 12:05:01 myLuna authpriv crit account configuration [preauth] ssdh[1234]: fatal: Access denied for user admin by PAM 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Setting tty modes failed: Invalid argument 2012 Feb 29 12:05:01 myLuna token manipulation error authpriv crit ssdh[1234]: fatal: PAM: pam_chauthtok(): Authentication Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 47 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna an error authpriv info ssdh[1234]: syslogin_perform_logout: logout() returned 2012 Feb 29 12:05:01 myLuna authpriv info ssdh[1234]: Received SIGHUP; restarting. These messages indicate abnormal SSH activity. A future revision of Luna Monitoring will provide more details of what each message means. Privileged Services Facility Keyword authpriv Software Process See list that follows Log File secure The following privileged services log messages to the secure log file with a software process identifier if shown. • • • • • • useradd[pid] chage[pid] passwd usermod[pid] login groupadd[pid] Expected Log Messages useradd 2012 Feb 29 12:05:01 myLuna authpriv info useradd[1234]: new user: name=recover, UID=0, GID=0, home=/home/recover, shell=/usr/lunasa/bin/recover 2012 Feb 29 12:05:01 myLuna authpriv info useradd[1234]: new user: name= , UID=0, GID=0, home=/home/admin, shell=/usr/lunasa/lush/lush 2012 Feb 29 12:05:01 myLuna authpriv info home=/usr/local/mysql, shell=/sbin/nologin useradd[1234]: new user: name=mysql, UID=500, GID=500, These messages indicate that the Linux utility useradd(1) successfully created accounts for the identified user (e.g., recover, admin, monitor, operatory or mysql). chage 2012 Feb 29 12:05:01 myLuna authpriv info chage[1234]: changed password expiry for This message indicates that the Linux utility chage(1) successfully changed the number of days between password changes and the date of the last password change for . is one of “admin”, “operator”, “monitor” or a user created by an administrator. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 48 of 102 Luna Monitoring Syslog & SNMP passwd 2012 Feb 29 12:05:01 myLuna for admin authpriv notice passwd: pam_unix(passwd:chauthtok): password changed This message indicates that the Linux utility passwd(1) successfully updated the admin user’s authentication token. usermod 2012 Feb 29 12:05:01 myLuna password authpriv info authpriv info usermod[1234]: change user ` ' This message indicates that the Linux utility usermod(1) successfully updated the login information for . is one of “admin”, “operator”, “monitor” or “audit.” login 2012 Feb 29 12:05:01 myLuna for user < admin | recover> authpriv authpriv info by LOGIN(uid=0) login: pam_unix(login:session): session opened 2012 Feb 29 12:05:01 myLuna for user authpriv authpriv info login: pam_unix(login:session): session closed 2012 Feb 29 12:05:01 myLuna authpriv authpriv info login: DIALUP AT ttyS0 BY The first two messages indicate that the Linux utility login(1) successfully established a new session with the Luna appliance or terminated a session. The third message indicates that the session is via the serial port on the front console of the Luna appliance. groupadd 2012 Feb 29 12:05:01 myLuna GID=<14 | 500> authpriv authpriv info groupadd[2558]: new group: name= , This message indicates that the Linux utility groupadd(1) successfully created a new group definition with the GID shown. The for uucp is 14; for mysql, 500. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. login 2012 Feb 29 12:05:01 myLuna unknown authpriv authpriv alert login: pam_unix(login:auth): check pass; user 2012 Feb 29 12:05:01 myLuna authpriv authpriv notice login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=ttyS0 ruser= rhost=192.168.0.100 2012 Feb 29 12:05:01 myLuna authpriv authpriv crit retrieving information about user login: pam_succeed_if(login:auth): error 2012 Feb 29 12:05:01 myLuna authpriv authpriv notice login: FAILED LOGIN 1 FROM (null) FOR , User not known to the underlying authentication module Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 49 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna retries; 4 > 3 authpriv authpriv alert 2012 Feb 29 12:05:01 myLuna authpriv authpriv err login: PAM service(login) ignoring max login: Authentication failure These messages indicate failure on the part of an administrator to login to the Luna appliance. The first four messages indicate that the login attempt was with a username unknown to the Luna appliance. The fifth message indicates that the threshold number of failed login attempts has been reached or exceeded. The last message is the Luna IS-specific message in place of the second message above. Kernel Facility Keyword kern Software Process kernel Log File messages Expected Log Messages Shutting Down Watchdog Timer 2012 Feb 29 12:05:01 myLuna kern info kernel: iTCO_wdt: Unexpected close, not stopping watchdog. 2012 Feb 29 12:05:01 myLuna kern info kernel: iTCO_wdt: Watchdog Module Unloaded. These messages indicate that the kernel shutdown the hardware watchdog timer, likely a result of the Lush “sysconf appliance watchdog disable” command. SafeNet Components SafeNet has developed a number of services that run within the Luna appliance. Some of these services use syslog to record significant actions and events. The following list identifies the SafeNet services that use syslog. • • • • • • • • • • sysstat – system status and monitoring (including LCD) cbs – HSM callback service (and remote PED) oamp – automatic monitoring of environment, detecting and determining faults ntls – Network trust link service htl – Host trust link service nbs – Network bus service rbs – Remote backup service Luna Administrative Shell (Lush) cron and crontab Certmonitord – NTLS certificate expiration monitoring Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 50 of 102 Luna Monitoring Syslog & SNMP sysstat The system status daemon reports on the health of the Luna appliance via syslog and the LCD on the front panel of the appliance. Facility Keyword user Software Process sysstatd Log File messages Expected Log Messages The following log messages are normal and expected entries in the log files. Service Started 2012 Feb 29 12:05:01 myLuna user info sysstatd: ------------ Luna System State Server - Started. This message indicates that the system status daemon is running, normal for a Luna appliance power up operation. Service Stopped 2012 Feb 29 12:05:01 myLuna user info sysstatd: ------------ Luna System State Server - Stopped. This message indicates that the system status daemon is stopped, normal for a Luna appliance shutdown operation. System Status 2012 Feb 29 12:05:01 myLuna user info sysstatd: Luna System State Server - OOS Errors: 15,100,60! 2012 Feb 29 12:05:01 myLuna 50,15,20,100,55,60! user info sysstatd: Luna System State Server - OFL Errors: 2012 Feb 29 12:05:01 myLuna user info sysstatd: Luna System State Server - InSrvTrb Errors: 1! Messages formatted like these examples represent normal reporting of Luna appliance health when it is not “in service.” The specific error codes indicate one or more potential problems with specific services of the Luna appliance operation. For details of their meaning, refer to the description of the error codes in the section “Front-panel Display” of the Luna product documentation. Up Time 2012 Feb 29 04:02:01 myLuna 0.38, 0.14, 0.04 syslog info logger: uptime: 04:02:01 up 6:50, 2 users, load average: This message indicates the uptime for the Luna appliance. cron is responsible for the message on a daily basis, usually early in the morning. Note that the facility keyword and software process are not the same as those for the system status daemon but is included in this section given its relationship to status monitoring. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 51 of 102 Luna Monitoring Syslog & SNMP Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Cannot Open Files Needed to Identify Product 2012 Feb 29 12:05:01 myLuna user err sysstatd: Cannot open PRODUCT file. 2012 Feb 29 12:05:01 myLuna user err sysstatd: Cannot open VERSION file. 2012 Feb 29 12:05:01 myLuna user err sysstatd: Cannot open RELEASE file. These messages indicate a missing or corrupted file necessary to identify the product on the LCD and/or via a Lush command. A PRODUCT:VERSION:RELEASE example is SA:5.3.0:5. Cannot Set Signal Handler 2012 Feb 29 12:05:01 myLuna user info sysstatd: Cannot set sighandler for SIGTERM. 2012 Feb 29 12:05:01 myLuna user info sysstatd: Cannot set sighandler for SIGINT. These messages indicate that the system status daemon was unable to set up signal handling for the SIGTERM and/or SIGINT signals. Failed to Detach 2012 Feb 29 12:05:01 myLuna user err sysstatd: Failed to detach sysstatd from console. This message indicates that the startup procedure for the system status daemon failed, specifically that the daemon did not launch into a background process. Unknown Error 2012 Feb 29 12:05:01 myLuna state. user info sysstatd: unknown error while trying to obtain the system This message indicates that the system status daemon was unable to determine health of the Luna appliance and represents a logic error within how the daemon was instrumented. Failed to Open Device 2012 Feb 29 12:05:01 myLuna user info sysstatd: Failed to open the LCD device. This message indicates that the system status daemon was unable to open the TTY device that maps to the LCD on the front of the Luna appliance. Failed to Open or Read File 2012 Feb 29 12:05:01 myLuna user info sysstatd: here:/usr/lunasa/sysstat/statuserror.txt. Did NOT find a system state error file 2012 Feb 29 12:05:01 myLuna user info /usr/lunasa/sysstat/statusoutput.txt. Did NOT find a system status file here: sysstatd: 2012 Feb 29 12:05:01 myLuna user info sysstatd: The SysStat service failed while reading the file: /usr/lunasa/sysstat/statusoutput.txt. It appears to be empty. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 52 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna user info sysstatd: The SysStat service cannot determine the system state. Missing the status script: /usr/lunasa/sysstat/statusscript.sh. 2012 Feb 29 12:05:01 myLuna user info sysstatd: the file:/usr/lunasa/sysstat/ethip.txt. The SysStat service failed while trying to open 2012 Feb 29 12:05:01 myLuna user info sysstatd: The SysStat service failed while reading the file: /usr/lunasa/sysstat/ethip.txt. It appears to be empty. These messages indicate a failure to find and/or open status files used by the system state daemon to determine the health of the Luna appliance. statuserror.txt shows the error codes, status codes and explanations for the different health status items reported for the Luna appliance. statusoutput.txt contains the current status of the Luna appliance. statusscript.sh is a shell script that creates the statusoutput.txt file by running several other scripts and utilities to collect the state of the Luna appliance. ethip.txt contains the details of the Ethernet ports (whether configured, IP address, host name and link status). Failed to Determine IP Address 2012 Feb 29 12:05:01 myLuna the ethernet IP addresses. user info sysstatd: The SysStat service failed while trying to obtain This message indicates that the script used to determine the IP address of the Luna appliance failed to execute successfully. HSM Callback Service The callback service is a SafeNet daemon that communicates with the cryptographic keycard (i.e., the PCI-e expansion card) within the Luna appliance. This service supports audit logging and remote PED features. Facility Keyword local5 Software Process Luna PED Client [pid] Log File lunalogs There are hundreds of callback service messages. A future version of Luna Monitoring will attempt to document callback service messages. If you encounter any and you would like more information, please contact SafeNet technical support for assistance. OAMP The oamp process monitors the cryptographic keycard and cryptographic operations within the Luna appliance. Facility Keyword local5 1 Software Process oamp[pid] Log File lunalogs1 See Database Service for details of messages in lunalogs. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 53 of 102 Luna Monitoring Syslog & SNMP 2 Facility Keyword Software Process local6 oamp[pid] A previous section in Luna Monitoring details log messages to hsm.log. Facility Keyword user Software Process logger Log File hsm.log2 Log File messages Expected Log Messages The following log messages are normal and expected entries in the log files. OAMP Started 2012 Feb 29 12:05:01 myLuna user notice logger: oamp started. This message indicates that the oamp process started successfully. Unexpected Log Messages There are no unexpected log messages. If you find any other entries in messages with the facility keyword user and software process logger, please contact SafeNet technical support to report the message and seek guidance on what to do next. Network Trust Link Service The network trust link service – NTLS – is the cryptographic services dispatcher of the Luna appliance. On system start up, NTLS runs as a daemon and listens for incoming connection requests on TCP/IP sockets. Using Secure Sockets Layer, NTLS secures communication between the appliance and a client, each party authenticating the other with certificates and encrypting data exchanged. SafeNet has architected NTLS into two principal components and the organization of NTLS log messages in this document reflects this architecture. A generic datapath engine manages the socket communications. This engine is common across Luna appliances (e.g., Luna SA, Luna SP and Luna IS use the same engine). A product-specific command processor parses and responds to the messages NTLS receives from and sends back to the client. The datapath engine underwent significant change for the Luna SA 5.2.2 release with many syslog message changes. This document makes explicit note for older messages; otherwise, assume that the descriptions apply for more current product releases. Facility Keyword Local5 Software Process NTLS [pid] Log File lunalogs Datapath: Expected Log Messages The following log messages are normal and expected entries in the log files from the datapath engine of NTLS. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 54 of 102 Luna Monitoring Syslog & SNMP Startup 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Starting up NTLS........ 2012 Feb 29 12:05:01 myLuna loaded local5 info NTLS [1234]: info : 0 : luna_engine_preload: SUCCESSFULLY 2012 Feb 29 12:05:01 myLuna local5 info ENGINE_ctrl_cmd_string(...SO_PATH... NTLS [1234]: info : 0 : 1 = 2012 Feb 29 12:05:01 myLuna local5 info ENGINE_ctrl_cmd_string(...ID...) NTLS [1234]: info : 0 : 1 = 2012 Feb 29 12:05:01 myLuna local5 info ENGINE_ctrl_cmd_string(...LOAD... NTLS [1234]: info : 0 : 1 = 2012 Feb 29 12:05:01 myLuna enabled for ALL ops local5 info NTLS [1234]: info : 0 : luna_engine_load: SUCCESSFULLY 2012 Feb 29 12:05:01 myLuna SSL operations enabled 3 local5 info NTLS [1234]: info : 0 : HSM crypto support for data path 2 2012 Feb 29 12:05:01 myLuna module version 1.0 loaded local5 info NTLS [1234]: info : 0 : "Luna SA 5.0 Command Processor" 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : NTLS data path configured with 5 inbound worker threads and 5 outbound worker threads 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : NTLS is online and operational. 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Data path TCP keep alive is configured as : TCP_KEEPIDLE = 10 sec : TCP_KEEPINTVL = 10 sec : TCP_KEEPCNT = 2 retries 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Data path TCP keep alive is configured as : TCP_KEEPIDLE = 10 sec : SSL cipher list set to AES256-GCM-SHA384 The first message indicates that NTLS is starting up. The next five messages trace the progress of the SSL start up sequence. The remaining messages show the progress through to a successful startup. Failed to Enable SSL Engine for All Operations 2012 Feb 29 12:05:01 myLuna local5 info ENGINE_set_default for ALL ops NTLS [1234]: info : 0 : luna_engine_load: failed to This message indicates that the SSL engine was unable to initialize properly. Client Connections 2012 Feb 29 12:05:01 myLuna 192.168.0.100/40847 local5 info NTLS [1234]: info : 0 : Incoming connection request... : 2012 Feb 29 12:05:01 myLuna from: 192.168.0.100/40847 local5 info NTLS [1234]: info : 0xc0000002 : Connection accepted 2012 Feb 29 12:05:01 myLuna local5 info 192.168.0.100/40847, result: Success NTLS [1234]: info : 0xc0000002 : Handshake result from: 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0xc0000002 : NTLS Client "192.168.0.100" connected and authenticated : 192.168.0.100/40847. 2 “NTLS” in older implementations. Or “disabled” or “NOT enabled” if the keys-in-hardware option is enabled but the SSL engine failed to load (“luna_engine_load: failed to ENGINE_set_default for ALL ops”). 3 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 55 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna 192.168.0.100/40847 local5 info 2012 Feb 29 12:05:01 myLuna local5 info disconnected: 192.168.0.100/40847 NTLS [1234]: info : 0xc0000002 : Connection terminating: NTLS [1234]: info : 0 : NTLA client "192.168.0.100" has 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Receive timer expired for client "192.168.0.100". Terminate client connection: 192.168.0.100/40847 4 2012 Feb 29 12:05:01 myLuna local5 info "192.168.0.100" : 192.168.0.100/40847 5 NTLS [1234]: info : 0 : Receive timer expired for client These messages indicate connect establishment and termination by NTLS. You should find pairs of connection accepted and disconnected messages in the logs. The last two messages result when a client connects but is inactive for a period configured for NTLS. Client Credentials Cannot be Verified 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0xc0000002 : Client credentials cannot be verified. Connection terminated : 192.168.0.100/40847 This message indicates that the client connection request could not be completed. A possible root cause is that the client’s certificate is invalid (e.g., has a wrong IP address or hostname). Cache Monitor 2012 Feb 29 12:05:01 myLuna location monitor started local5 info NTLS [1234]: info : 0 : Client certificate verify 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : removed from verify location. Flush verify location cache 0 : Detected client certificate 2012 Feb 29 12:05:01 myLuna location monitor terminated 0 : Client certificate verify local5 info NTLS [1234]: info : These messages indicate that NTLS has started to monitor client certificates registered with the process. Connection Instance Added and Removed 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : Connection instance removed : 192.168.0.100/40847 0 : NTLA Client "192.168.0.100" 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : connected and authenticated : 192.168.0.100/40847 0 : NTLA Client "192.168.0.100" 2012 Feb 29 12:05:01 myLuna local5 info disconnected: 192.168.0.100/40847 NTLS [1234]: info : 0 : NTLA Client "192.168.0.100" has 2012 Feb 29 12:05:01 myLuna local5 info disconnected: 192.168.0.100/40847 NTLS [1234]: info : 0 : NTLA Client "192.168.0.100" has These messages indicate NTLS’s action to add and remove connection instances. 4 5 This message results if the timeout is for a scheduled cleanup. This message results if the timeout is not for a scheduled cleanup. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 56 of 102 Luna Monitoring Syslog & SNMP Client Credentials Cannot be Verified 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : verified. Connection terminated : 192.168.0.100/40847 0 : Client credentials cannot be This message indicates that an application connected to NTLS but did not provide the SSL credentials to complete authentication. Handshake Failed 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0xc0000711 : Fail to establish a secure channel with client : 192.168.0.100/40847 : RC_SSL_FAILED_HANDSHAKE This message indicates that an application attempted to connect with SSL credentials that did not conform to the algorithms and/or cryptographic strength expected by NTLS. Unable to Initialize Server Socket 2012 Feb 29 12:05:01 myLuna local5 crit could not Bind : RC_SOCKET_ERROR NTLS [1234]: critical : 0xc0000500 : NTLS listening port 2012 Feb 29 12:05:01 myLuna local5 err path 6 server socket : RC_SOCKET_ERROR NTLS [1234]: error : 0xc0000500 : Unable to initialize data 2012 Feb 29 12:05:01 myLuna failed : RC_SOCKET_ERROR NTLS [1234]: error : 0xc0000500 : NTLS initialization local5 err These messages indicate that NTLS was unable to initialize the server socket. A possible root cause is that the network interface is not properly configured. Connected to Server 2012 Feb 29 12:05:01 myLuna local5 info to server : 192.168.0.100/40847 NTLS [1234]: info : 0xc0000002 : 192.168.0.100 connected This message indicates that NTLS has successfully established an outgoing connection with a peer or remote server. Shutting Down 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Shutting down NTLS.... 2012 Feb 29 12:05:01 myLuna termination. local5 info NTLS [1234]: info : 0 : Intiating [sic] dispatch 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Dispatching has been terminated. 2012 Feb 29 12:05:01 myLuna local5 info 7 2012 Feb 29 12:05:01 myLuna shut downed local5 info NTLS [1234]: info : 0 : "Luna SA 5.0 Command Processor" 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : NTLS terminated 6 7 NTLS [1234]: info : 0 : Terminating. “NTLS” in older implementations. For Luna SA 5.2.2 and Luna SA 5.3 releases, the severity level is “critical” but is “info” in the Luna SA 5.4 release. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 57 of 102 Luna Monitoring Syslog & SNMP These messages show the progression of steps as NTLS terminates. Datapath: Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please consult the user documentation to seek guidance on how to correct the problem. NTLS Terminating 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000102 : Terminating. 2012 Feb 29 12:05:01 myLuna daemon. Terminating. local5 crit NTLS [1234]: critical : 0xc0000102 : Failed to start as 2012 Feb 29 12:05:01 myLuna application. Terminating. local5 crit NTLS [1234]: critical : 0xc0000102 : Failed to initialize 2012 Feb 29 12:05:01 myLuna server. Terminating. local5 crit NTLS [1234]: critical : 0xc0000102 : Failed to create These messages indicate that NTLS failed at startup. Possible root causes for the first message are a failure to create internal data structures needed by NTLS or a failure to generate the private key and certificate needed by NTLS. Failed to Load SSL Engine 2012 Feb 29 12:05:01 myLuna ENGINE_by_id local5 info NTLS [1234]: info : 0 : luna_engine_load: failed to This message indicates that OpenSSL was unable to load the specified engine. A possible root cause is the wrong engine identifier specified in the OpenSSL configuration file. Skip C_Initialize 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : 1 = ENGINE_ctrl_cmd_string(...SKIP_C_INITIALIZE...) This message indicates that the OpenSSL engine for LunaCA3 skipped the call the C_Initialize . Luna appliances do not use the LunaCA3 engine. Configured for RSA Operations 2012 Feb 29 12:05:01 myLuna enabled for RSA ops local5 info NTLS [1234]: info : 0 : luna_engine_load: SUCCESSFULLY This message indicates that the SSL engine used by NTLS is configured only for RSA operations. Failed to Pre-Load SSL Engine Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : luna_engine_preload: failed to load This message indicates that the SSL engine was unable to preload. A possible root cause is the wrong engine identifier configured on the Luna appliance. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 58 of 102 Luna Monitoring Syslog & SNMP Failed to Enable SSL Engine for RSA Operations 2012 Feb 29 12:05:01 myLuna local5 info ENGINE_set_default for RSA ops NTLS [1234]: info : 0 : luna_engine_load: failed to This message indicates that the SSL engine was unable to initialize properly for RSA operations. Failed to Initialize 2012 Feb 29 12:05:01 myLuna for NTLS. local5 info NTLS [1234]: info : 0 : Missing command processor library 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000102 : Command processor module not found : "Luna SA 5.0 Command Processor" : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_get_supported_versions function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_initialize_p function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_get_processor_name_p function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_shutdown function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_client_registration_observer function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000105 : cp_configure function not found in command processor "Luna SA 5.0 Command Processor" : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Failed to initialize "Luna SA 5.0 Command Processor" module version 1.0 for data path module version 1.0 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000102 : Failed to initialize "Luna SA 5.0 Command Processor" module version 1.0 : RC_GENERAL_ERROR 8 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000105 : "Luna SA 5.0 Command Processor" module version 1.0 is incompatable [sic] with data path : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 err openssl library : RC_SOCKET_ERROR NTLS [1234]: error : 0xc0000500 : Unable to initialize 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000500 : Unable to initialize openssl server context : RC_SOCKET_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit initialization failed for NTLS. NTLS [1234]: critical : 0xc0000102 : Application specific 2012 Feb 29 12:05:01 myLuna dispatcher. NTLS [1234]: critical : 0xc0000102 : Unable to initialize local5 crit 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration client certificate file name invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration client private key file name invalid: RC_FILE_NAME_INVALID 8 For older implementations. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 59 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration ssl configuration file name invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : private key/certificate: RC_GENERAL_ERROR, error: -1 0xc0000102 : Failed to create These messages indicate a failure of the datapath component to initialize itself at NTLS startup. No Command Processor Module Configured 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Data path is not configured with a command processor module. Use default command processor setting This message indicates an error in how NTLS is configured. Fail to Accept Connections 2012 Feb 29 12:05:01 myLuna connection. local5 err NTLS [1234]: error : 0xc0000002 : Unable to create a new 2012 Feb 29 12:05:01 myLuna RC_GENERAL_ERROR local5 err NTLS [1234]: error : 0xc0000002 : Accept failed. Reason: These messages indicate an NTLS failure to establish the TCP/IP socket necessary to listen for and accept connections from clients. Fail to Create Client Connection 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000500 : Fail to create connection instance : 192.168.0.100/40847 : RC_SOCKET_ERROR 2012 Feb 29 12:05:01 myLuna local5 warn client TCP connection : RC_SOCKET_ERROR NTLS [1234]: warning : 0xc0000500 : Fail to accept a These messages indicate a failure to create a client connection on an incoming request. Cache Monitor Failed 2012 Feb 29 12:05:01 myLuna local5 err initialize : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 : CA cache monitor fail to This message indicates that the thread spawned to monitor client certificates failed to initialize. Connection Instance Failures 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to create timer object for connection instance : 192.168.0.100/40847 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000004 : Fail to create ssl object for connection instance : 192.168.0.100/40847 : RC_UNEXPECTED 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : header malformed : 192.168.0.100/40847 : RC_DATA_INVALID 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : buffer : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 0xc0000102 : Client request 0xc0000001 : Fail to allocate command 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to forward client request to command processor module : 192.168.0.100/40847 : RC_GENERAL_ERROR Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 60 of 102 Luna Monitoring Syslog & SNMP These messages indicate a failure of NTLS to add or remove a client connection instance. Non-Luna Client Connect Attempt 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : non-Luna client : 192.168.0.100/40847 : RC_UNEXPECTED 0xc0000004 : Connection attempt from This message indicates that an application attempted to connect to the Luna appliance via NTLS but did not provide the proper SSL message exchange expected. Command Process Unloaded 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Command Processor module unloaded This message indicates that NTLS unloaded its command processor module (a shared object library). SSL Certificates Problems 2012 Feb 29 12:05:01 myLuna local5 crit determining hostname: RC_SSL_CTX_ERROR NTLS [1234]: critical : 0xc0000701 : Bad SSL handle while 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc000070a : Failed to get current certificate while determining hostname: RC_SSL_INVALID_CERT_STRUCTURE 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc000070a : Failed to get certificate text while determining hostname: RC_SSL_INVALID_CERT_STRUCTURE 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc000070a : Invalid client certificate while determining hostname: RC_SSL_INVALID_CERT_STRUCTURE 2012 Feb 29 12:05:01 myLuna local5 crit native SSL CTX handle: RC_SSL_CTX_ERROR NTLS [1234]: critical : 0xc0000701 : Unable to retrieve 2012 Feb 29 12:05:01 myLuna local5 crit cipher list: RC_SSL_CIPHER_LIST_ERROR NTLS [1234]: critical : 0xc0000702 : SSL error setting 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration client certificate path invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000703 : Load verify locations failed for: CACert.pem, RC_SSL_CERT_VERIFICATION_LOCATION_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000703 : Load verify locations failed for single client: CACert.pem, RC_SSL_CERT_VERIFICATION_LOCATION_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration server certificate filename invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000704 : Chrystoki using server certificate failed: /usr/lunasa/vts/server/server.pem, RC_SSL_LOAD_SERVER_CERT_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration server private key filename invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : server private key: RC_SSL_LOAD_SERVER_PRIVATE_KEY_ERROR 0xc0000705 : Error loading the 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : server private key: RC_SSL_VALIDATE_SERVER_PRIVATE_KEY_ERROR 0xc0000706 : Error checking the 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : host IP Address : RC_SOCKET_ADDRESS_INVALID 0xc0000406 : Fail to retrieve Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 61 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 crit host port : RC_SOCKET_ADDRESS_INVALID NTLS [1234]: critical : 0xc0000406 : Fail to retrieve 2012 Feb 29 12:05:01 myLuna local5 crit context. : RC_SOCKET_ADDRESS_INVALID NTLS [1234]: critical : 0xc0000406 : Unable to create 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : server for address 192.168.0.100/40847 : RC_GENERAL_ERROR 0xc0000002 : Failed to initialize 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000002 : Chrystoki configuration client certificate filename invalid: RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000708 : Chrystoki using client certificate failed: CACert.pem, RC_SSL_LOAD_CLIENT_CERT_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration client private key filename invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : client private key: RC_SSL_LOAD_CLIENT_PRIVATE_KEY_ERROR 0xc000070b : Error loading the 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : 0xc0000402 : Chrystoki configuration server certificate authority filename invalid: RC_FILE_NAME_INVALID 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : certificate authorities: RC_SSL_CERT_VERIFICATION_LOCATION_ERROR 0xc0000703 : Error loading server 2012 Feb 29 12:05:01 myLuna context. : RC_GENERAL_ERROR 0xc0000002 : Unable to update local5 crit NTLS [1234]: critical : These messages indicate a failure of the underlying communication mechanism NTLS uses to exchange messages between a client application and the Luna appliance. Client Authentication Failures 2012 Feb 29 12:05:01 myLuna local5 info string. Reason: RC_GENERAL_ERROR NTLS [1234]: info : 0xc0000002 : Failed writing identity 2012 Feb 29 12:05:01 myLuna RC_GENERAL_ERROR local5 info NTLS [1234]: info : 0xc0000002 : Failed connect. Reason: 2012 Feb 29 12:05:01 myLuna Reason: RC_GENERAL_ERROR local5 info NTLS [1234]: info : 0xc0000002 : Failed handshake. 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 192.168.0.100/40847, result: RC_GENERAL_ERROR 0xc0000002 : Identify result from: 2012 Feb 29 12:05:01 myLuna local5 info string. Reason: RC_GENERAL_ERROR 0xc0000002 : Failed reading identity NTLS [1234]: info : 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to forward dataless client request to command processor module : 192.168.0.100/40847 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : header malformed : 192.168.0.100/40847 : RC_DATA_INVALID 0xc0000102 : Client request 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : command buffer : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 0xc0000001 : Fail to allocate 2012 Feb 29 12:05:01 myLuna local5 info buffer. Reason: RC_GENERAL_ERROR NTLS [1234]: info : 0xc0000002 : Failed reading command 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to forward client request to command processor module : 192.168.0.100/40847 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna Reason: RC_GENERAL_ERROR local5 info Luna Syslog and SNMP Monitoring Guide NTLS [1234]: info : 0xc0000002 : Failed writing to client 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 62 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : reinitializing failed. Reason: RC_GENERAL_ERROR 0xc0000002 : Last write before 2012 Feb 29 12:05:01 myLuna local5 info operation, actual: RC_GENERAL_ERROR NTLS [1234]: info : 0xc0000002 : Expected cancelled 2012 Feb 29 12:05:01 myLuna local5 info connection for: 192.168.0.100/40847 NTLS [1234]: info : 0xc0000002 : Reinitializing 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : buffer: 192.168.0.100/40847 : RC_GENERAL_ERROR 0xc0000002 : Fail to get response 2012 Feb 29 12:05:01 myLuna certificate. None received. NTLS [1234]: error : 0xc0000002 : Expecting client 2012 Feb 29 12:05:01 myLuna local5 info client certificate.: 192.168.0.100/40847 NTLS [1234]: info : 0xc0000002 : Unable to retrieve 2012 Feb 29 12:05:01 myLuna local5 info failed : 192.168.0.100/40847 NTLS [1234]: info : 0xc0000002 : cp_server_connected local5 err These messages indicate a failure of NTLS to verify the client requesting to connect to the Luna appliance. Port Monitoring Failures 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: critical : host IP Address : RC_SOCKET_ADDRESS_INVALID 0xc0000406 : Fail to retrieve 2012 Feb 29 12:05:01 myLuna local5 crit host port : RC_SOCKET_ADDRESS_INVALID NTLS [1234]: critical : 0xc0000406 : Fail to retrieve 2012 Feb 29 12:05:01 myLuna local5 crit linger option : RC_SOCKET_ERROR NTLS [1234]: critical : 0xc0000500 : Fail to set socket 2012 Feb 29 12:05:01 myLuna local5 crit reuse option : RC_SOCKET_ERROR NTLS [1234]: critical : 0xc0000500 : Fail to set socket 2012 Feb 29 12:05:01 myLuna local5 crit port could not Bind : RC_SOCKET_ERROR NTLS [1234]: critical : 0xc0000500 : Data path listening 2012 Feb 29 12:05:01 myLuna local5 crit non-blocking option : RC_SOCKET_ERROR NTLS [1234]: critical : 0xc0000500 : Fail to set socket 2012 Feb 29 12:05:01 myLuna port not listening NTLS [1234]: critical : 0xc0000002 : Data path listening local5 crit These messages indicate a failure of NTLS to set up the port upon which the process will listen for incoming connection requests. Could Not Initialize Command Processor 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Failed to initialize "Luna SA 5.0 Command Processor" module version 1.0 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err command processor cleanup mechanism. NTLS [1234]: error : 0xc0000002 : Failed to initialize 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Failed to create Appliance Handler for NTLS: RC_GENERAL_ERROR. 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Failed to initialize Appliance Handler for NTLS: RC_GENERAL_ERROR. 2012 Feb 29 12:05:01 myLuna for NTLS: RC_GENERAL_ERROR. local5 err Luna Syslog and SNMP Monitoring Guide NTLS [1234]: error : 0xc0000002 : Failed to create server 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 63 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 err pool for NTLS: RC_GENERAL_ERROR. NTLS [1234]: error : 0xc0000002 : Failed to create thread These messages indicate a failure to initialize the command processor, a shared library module within the Luna appliance. Datapath Debug Messages 2012 Feb 29 12:05:01 myLuna 0.9.5a 1 Apr 2000 local5 debug NTLS [1234]: debug : 0 : Data path is using OpenSSL 2012 Feb 29 12:05:01 myLuna response in local5 debug NTLS [1234]: debug : 0xc0000002 Unexpected connect 2012 Feb 29 12:05:01 myLuna response in local5 debug NTLS [1234]: debug : 0xc0000002 Unexpected client 2012 Feb 29 12:05:01 myLuna response in local5 debug NTLS [1234]: debug : 0xc0000002 Unexpected handshake 2012 Feb 29 12:05:01 myLuna response in local5 debug NTLS [1234]: debug : 0xc0000002 Unexpected read 2012 Feb 29 12:05:01 myLuna response in local5 debug NTLS [1234]: debug : 0xc0000002 Unexpected write SafeNet does not enable logging of debug messages for NTLS. If in future SafeNet does enable this level of log messages, the messages above show examples of what you might find in the log files. In the examples above, is one of the following set: [IDENTITY STATE | HANDSHAKE STATE | PROCESSING STATE | HEADER STATE | COMMAND STATE | CONNECTING STATE | IDENTIFYING STATE | OUT OF SERVICE STATE | REINITIALIZING STATE]. NTLS Crash and Burn 2012 Feb 29 12:05:01 myLuna local5 crit NTLS [1234]: info : 0 : NTLS CRASH AND BURN! Stack dump saved to /var/log/ntls_bt_2012-02-29_12:05:01 2012 Feb 29 12:05:01 myLuna dump the stack! local5 crit NTLS [1234]: info : 0 : NTLS CRASH AND BURN and unable to These messages indicate a programming error. The first message indicates that NTLS terminated abnormally (on one of SIGSEGV, SIGILL or SIGBUS signals), generating a stack trace file in the process. An example stack trace file is: ntls_bt_2012-02-29_12:05:01 found in the logs directory. Forwarding this file to SafeNet product engineering may assist a developer to isolate the reason for the abnormal termination. The second message indicates an abnormal termination but with no resulting stack trace created. Luna SA Command Processor: Expected Log Messages The following log messages are normal and expected entries in the log files from the Luna SA command processor of NTLS. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 64 of 102 Luna Monitoring Syslog & SNMP Configured with Worker Threads 2012 Feb 29 12:05:01 myLuna with 50 worker threads local5 info NTLS [1234]: info : 0 : SA command processor configured This message indicates that NTLS has successfully started its command processor. HTL Configured 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: info : 0 : Listening for HTL kill requests This message indicates that host trust link (HTL) is configured and started. USB Backup and PKI Bundle Device 2012 Feb 29 12:05:01 myLuna has been undeployed! local5 info NTLS [1234]: USB token: Backup1 with serial number: 123456 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: Re-deployed the serial number: 123456 to Virtual token list USB token: Backup1 with 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: Added USB token: Backup1 with serial number: 123456 at slot: 4 and container id: 8 to Virtual token list 2012 Feb 29 12:05:01 myLuna local5 info 123456 inserted into slot 4! NTLS [1234]: Deployed token Backup1 with Serial Number 2012 Feb 29 12:05:01 myLuna local5 info 123456 inserted into slot 4! NTLS [1234]: NON Deployed token Backup1 with Serial Number 2012 Feb 29 12:05:01 myLuna local5 info Number 123456 inserted into slot 4! NTLS [1234]: NON Deployed token with no label and Serial 2012 Feb 29 12:05:01 myLuna 123456 ejected from slot 4! local5 info NTLS [1234]: Deployed token Backup1 with Serial Number 2012 Feb 29 12:05:01 myLuna 123456 ejected from slot 4! local5 info NTLS [1234]: NON Deployed token Backup1 with Serial Number 2012 Feb 29 12:05:01 myLuna local5 info Number 123456 ejected from slot 4! NTLS [1234]: NON Deployed token with no label and Serial 2012 Feb 29 12:05:01 myLuna local5 info with serial number: 123456 at slot: 4 NTLS [1234]: info : 0 : Waited to find PKI token: PKIToken These messages indicate actions associated with devices connected to the USB port on the Luna appliance. Luna SA supports secure backup from the internal HSM to a USB-attached backup device. Luna SA also supports a “PKI bundle” feature for root keys stored and accessed on a USB-attached device (e.g., Luna G5). The examples above show “Backup1” for the label name of the USB-attached device but this label could be PKI bundle-centric (e.g., “RootCA1”) – the messages are common for both use cases. Command Processor Instance Removed 2012 Feb 29 12:05:01 myLuna local5 info NTLS [1234]: Command processor instance for client 192.168.0.100 removed : 192.168.0.100/40847 This message indicates that a client connection terminated and NTLS removed the command processor instance allocated for this client. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 65 of 102 Luna Monitoring Syslog & SNMP HTL Kill Request 2012 Feb 29 12:05:01 myLuna local5 info HTL for client 192.168.0.100 NTLS [1234]: info : 0 : Received NTLS kill request from This message indicates that the host trust link was forcibly terminated for the specified client. Shutdown 2012 Feb 29 12:05:01 myLuna shut downed local5 info NTLS [1234]: info : 0 : "Luna SA 5.0 Command Processor" This message indicates an orderly shutdown of the Luna SA command processor. Luna SA Command Processor: Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please consult the user documentation to seek guidance on how to correct the problem. Keep-alive Timer Failures 2012 Feb 29 12:05:01 myLuna local5 warn NTLS [1234]: warning : 0xc0000002 : Fail to create client command keepalive timer : 192.168.0.100/40847 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 warn NTLS [1234]: warning : 0 : Client command keepalive function is disabled : 192.168.0.100/40847 These messages indicate a failure to start the keep-alive timer that monitors for inactive client connections. Startup and Operational Failures 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate connection request response buffer [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate connection request response buffer [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate termination response buffer [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate slot state response buffer [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate response message buffer [type = 7] [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate response message buffer [type = 7] [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Fail to allocate a keep alive message buffer [size = 100] : 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err interface : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 Fail to initialize token 2012 Feb 29 12:05:01 myLuna local5 err mapping object : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 Fail to initialize App ID 2012 Feb 29 12:05:01 myLuna PED support. NTLS [1234]: error : 0xc0000002 Failed to initialize remote local5 err Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 66 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 Fail to initialize client registration database : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 Fail to populate client registration database : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 Fail to reset client registration entries flags : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err manager object : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 Fail to create token 2012 Feb 29 12:05:01 myLuna Manager : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 Fail to Initialize VToken local5 err 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000105 Version 5.0 is not supported by command processor version 1.0 : RC_FUNCTION_NOT_SUPPORTED 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Connection count is not incremented for APP ID [b43 : 0 : 1] : [5 : 1] 192.168.0.100/40847 :RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 : Out of Memory Error in ConnectionClass::AttachResponseBuffer SetResponseBuffer : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000001 Fail to create command processor instance for client 192.168.0.100 : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 2012 Feb 29 12:05:01 myLuna observer. local5 err NTLS [1234]: error : 0xc0000002 Invalid client registration 2012 Feb 29 12:05:01 myLuna observer operation: 9. local5 err NTLS [1234]: error : 0xc0000002 Invalid client registration 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0xc0000002 Fail to return a command response to 192.168.0.100 : 192.168.0.100/40847 : RC_GENERAL_ERROR These messages indicate a failure of NTLS to start the component responsible for managing connection queues and for processing HSM-specific requests for service from clients. Fail to Initialize Appliance Statistics 2012 Feb 29 12:05:01 myLuna local5 warn NTLS [1234]: warning : 0xc0000002 : Fail to initialize appliance statistic object : RC_GENERAL_ERROR This message indicates a failure of the Luna SA command processor to initialize data structures necessary to compile operational statistics. NTLS continues to operate. HTL Required 2012 Feb 29 12:05:01 myLuna HTL status is 5 local5 info NTLS [1234]: info : 0 : Client 192.168.0.100 requires HTL; This message indicates that the client is configured to use HTL but the host trust link is not up. HTL Failure 2012 Feb 29 12:05:01 myLuna local5 err NTLS [1234]: error : 0 : Exception in HTL kill listener: This message indicates a failure of the HTL component of the Luna SA 5 command processor. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 67 of 102 Luna Monitoring Syslog & SNMP Client Credentials Cannot be Verified 2012 Feb 29 12:05:01 myLuna local5 info verified : 192.168.0.100/40847 NTLS [1234]: info : 0 : Client credentials cannot be This message indicates that the client credentials cannot be verified. A possible root cause is an inconsistent entry in the client certificate for IP address or host name. Client Terminating 2012 Feb 29 12:05:01 myLuna local5 info termination : 192.168.0.100/40847 NTLS [1234]: info : 0 : Client 192.168.0.100 requested This message indicates that the client is terminating under normal (i.e., graceful) circumstances. Unsupported Command 2012 Feb 29 12:05:01 myLuna 0x00000000. local5 info NTLS [1234]: Client attempted unsupported command This message indicates that the command request received by NTLS is unexpected. A possible root cause is from an application that calls a PKCS#11 function that NTLS is not programmed to either accept or ignore. Failure to Retrieve Container List 2012 Feb 29 12:05:01 myLuna local5 err list : LUNA_RET_DEVICE_ERROR NTLS [1234]: error : 0x300000 : Fail to retrieve container 2012 Feb 29 12:05:01 myLuna local5 crit containers! : LUNA_RET_DEVICE_ERROR NTLS [1234]: critical : 0x300000 : Unable to load system These messages indicate that NTLS was unable to retrieve from the internal HSM its representation of how cryptographic objects are segregated from one another. A possible root cause is a loss of communication between the internal HSM and its associated device driver. Luna SA Command Processor Debug Messages 2012 Feb 29 12:05:01 myLuna local5 debug : 1] removed :192.168.0.100/40847 NTLS [1234]: debug : 0 : Transient APP ID Mapping [5 / 5 2012 Feb 29 12:05:01 myLuna local5 debug : [5 : 1] : 192.168.0.100/40847 NTLS [1234]: debug : 0 : Open session : [1 / 0 : 1 : 20] 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Open session APP ID major [b43] doesn't have 0x80000000 set : [5 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Create transient APP ID mapping for open session : [b43 : 0 : 1] : [5 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : APP ID mapping already exist : [b43 : 0 : 1] : [5 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug 0 : 1] : [5 : 1] : 192.168.0.100/40847 NTLS [1234]: debug : 0 : Open session get APP ID : [b43 : 2012 Feb 29 12:05:01 myLuna local5 debug 1] : [5 : 1] : 192.168.0.100/40847 NTLS [1234]: debug : 0 : Close all sessions : [b43 : 0 : Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 68 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Open Access : b43 : 0 : 1] :192.168.0.100/40847 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Create persistent APP ID mapping for [b43 : 0 : 1] : [5 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Change existing APP ID mapping for [5 : 1] to persistent state : [80000005 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna : 192.168.0.100/40847 local5 debug NTLS [1234]: debug : 0 : Closing Access for [b43 : 0 : 1] 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Close Access APP ID major [b43] doesn't have 0x80000000 set : [5 : 1] : 192.168.0.100/40847 2012 Feb 29 12:05:01 myLuna local5 debug [5 : 1] removed :192.168.0.100/40847 NTLS [1234]: debug : 0 : APP ID Mapping [b43 : 0 : 1] : 2012 Feb 29 12:05:01 myLuna local5 debug command : 192.168.0.100/40847 NTLS [1234]: debug : 0 : Bypassed LUNA_SEED_RANDOM 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Closed Session : [5 : 1 : 266] 2012 Feb 29 12:05:01 myLuna 1] local5 debug NTLS [1234]: debug : 0 : Closed All Sessions : [5 / 5 : 2012 Feb 29 12:05:01 myLuna 1 : 20] local5 debug NTLS [1234]: debug : 0 : Removed All Sessions : [5 / 5 : 2012 Feb 29 12:05:01 myLuna : 266] local5 debug NTLS [1234]: debug : 0 : Opened Session : [5 / 5 : 1 : 20 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Created VToken for 4/8. 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : viper slot = 1 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Found 20 partitions on HSM. 2012 Feb 29 12:05:01 myLuna S/N 123456 amount 4 tokens. local5 debug NTLS [1234]: debug : 0 : Looking for index for token with 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Token 4 has S/N 123456. 2012 Feb 29 12:05:01 myLuna local5 debug NTLS [1234]: debug : 0 : Token 4 is dirty. 2012 Feb 29 12:05:01 myLuna local5 debug USBSlotHandler::MonitorSlot for slot 4 NTLS [1234]: debug : 0 : Entering 2012 Feb 29 12:05:01 myLuna local5 debug USBSlotHandler::MonitorSlot for slot 4 NTLS [1234]: debug : 0 : Leaving 2012 Feb 29 12:05:01 myLuna local5 debug USBSlotHandler::MonitorConfig for slot 4 NTLS [1234]: debug : 0 : Entering 2012 Feb 29 12:05:01 myLuna local5 debug USBSlotHandler::MonitorConfig for slot 4 NTLS [1234]: debug : 0 : Leaving SafeNet does not enable logging of debug messages for NTLS. If in future SafeNet does enable this level of log messages, the messages above show examples of what you might find in the log files. These messages are for developer testing and hence are undocumented. Virtual Token Create and Add Failures 2012 Feb 29 12:05:01 myLuna to table : RC_GENERAL_ERROR local5 err NTLS [1234]: error : 0xc0000002 : Unable to add VToken 4/8 2012 Feb 29 12:05:01 myLuna 4/8 : RC_GENERAL_ERROR local5 err NTLS [1234]: error : 0xc0000002 : Unable to start VToken Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 69 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 err VToken 4/8 : RC_GENERAL_ERROR NTLS [1234]: error : 0xc0000002 : Unable to initialize These messages indicate a failure to complete an intended operation on a virtual token within NTLS. Host Trust Link Host Trust Link or HTL is an optional service to tightly bind communication between a client application and the Luna appliance. HTL offers protection of appliance/client registrations for cloud solutions. HTL operates as a process within the Luna appliance (htld). Facility Keyword Local5 Software Process HTLD[pid] Log File lunalogs Expected Log Messages Normal Startup 2012 Feb 29 12:05:01 myLuna slot 2 local5 info HTLD[1234]: info : 0 : NBSD loaded a PCI HSM at physical 2012 Feb 29 12:05:01 myLuna local5 info configured with 50 worker threads HTLD[1234]: info : 0 : Network Bus Command Processor 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : Listening for HTL status queries 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : Listening for OTT expiry events 2012 Feb 29 12:05:01 myLuna local5 info Processor" module version 2.0 loaded HTLD[1234]: info : 0 : "Luna Network Bus 1.0 Command These messages indicate normal startup of the HTL process. Client Connection 2012 Feb 29 12:05:01 myLuna 192.168.0.100/40847 local5 info HTLD[1234]: info : 0 : Connection accepted from: 2012 Feb 29 12:05:01 myLuna local5 info 192.168.0.100/40847, result: Success HTLD[1234]: info : 0 : Identify result from: 2012 Feb 29 12:05:01 myLuna local5 info 192.168.0.100/40847, result: Success HTLD[1234]: info : 0 : Handshake result from: 2012 Feb 29 12:05:01 myLuna HTLD[1234]: info : 0 : Starting connection processing local5 info 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : HTLD Client "192.168.0.100" connected and authenticated : 192.168.0.100/40847. 2012 Feb 29 12:05:01 myLuna successfully local5 info HTLD[1234]: info : 0 : One-time token validated 2012 Feb 29 12:05:01 myLuna private key sent to client local5 info HTLD[1234]: info : 0 : Sending dynamic certificate and 2012 Feb 29 12:05:01 myLuna 192.168.0.100/40847 local5 info HTLD[1234]: info : 0 : Reinitializing connection for: 2012 Feb 29 12:05:01 myLuna local5 info 192.168.0.100/40847, result: Success Luna Syslog and SNMP Monitoring Guide HTLD[1234]: info : 0 : Handshake result from: 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 70 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : Starting connection processing 2012 Feb 29 12:05:01 myLuna : 192.168.0.100/40847 local5 info HTLD[1234]: info : 0 : Client connection exists; resuming 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : HTLD Client "192.168.0.100" connected and authenticated : 192.168.0.100/40847. 2012 Feb 29 12:05:01 myLuna client local5 info HTLD[1234]: info : 0 : Initial counter information sent to 2012 Feb 29 12:05:01 myLuna HTL link is up local5 info HTLD[1234]: info : 0 : Client acknowledged counter data; These messages indicate successful HTL establishment between the HTL service and a client. Connection Removed 2012 Feb 29 12:05:01 myLuna 192.168.0.100; cleaning up local5 info HTLD[1234]: info : 0 : Grace period expired for client 2012 Feb 29 12:05:01 myLuna user 192.168.0.100 local5 info HTLD[1234]: info : 0 : Terminated 1 NTLS connections for 2012 Feb 29 12:05:01 myLuna local5 info HTLD[1234]: info : 0 : Command processor instance for client 192.168.0.100 removed : 192.168.0.100/40847 These messages indicate that the HTL service terminated a host trust link with a client. Instance Wrapped 2012 Feb 29 12:05:01 myLuna ID wrapped around. local5 info HTLD[1234]: info : 0 : Unique command processor instance This message indicates that the internal counter that maps HTL identifiers reached its limit and reset to 1. Shutdown 2012 Feb 29 12:05:01 myLuna local5 info 9 2012 Feb 29 12:05:01 myLuna Processor" shutting down. local5 info HTLD[1234]: info : 0 :: Terminating. HTLD[1234]: info : 0 : "Luna Network Bus 1.0 Command These messages indicate an orderly shutdown of the HTL service. Unexpected Log Messages Incompatible Components 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000105 : Version 2.0 is not supported by HTL command processor : RC_FUNCTION_NOT_SUPPORTED This message indicates an incompatibility between the datapath and command processor components of the HTL service. 9 On Luna SA 5.2.2 and Luna SA 5.3 releases, the severity is “critical” but this message is only informational. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 71 of 102 Luna Monitoring Syslog & SNMP Client Connection Failures 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000002 : Fail to return a command response to 192.168.0.100 : 192.168.0.100/40847 : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Fail to create command processor instance for client 192.168.0.100 : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 2012 Feb 29 12:05:01 myLuna registration observer. local5 err HTLD[1234]: error : 0xc0000002 : Invalid client 2012 Feb 29 12:05:01 myLuna local5 err registration observer operation: 9. HTLD[1234]: error : 0xc0000002 : Invalid client These messages indicate a failure of the HTL service to complete the host trust link for a client. Client Connection in Progress 2012 Feb 29 12:05:01 myLuna local5 info progress : 192.168.0.100/40847 HTLD[1234]: info : 0 : Client connection already in This message indicates that the HTL service detected a connection request in an unexpected state. Invalid Client Credentials 2012 Feb 29 12:05:01 myLuna local5 info verified : 192.168.0.100/40847 HTLD[1234]: info : 0 : Client credentials cannot be This message indicates that the HTL service did not receive valid client credentials. One-Time Token Expiry Handler 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0 : Exception in OTT expiry handler: 2012 Feb 29 12:05:01 myLuna local5 err expiry handler: HTLD[1234]: error : 0 : Error accepting connection in OTT 2012 Feb 29 12:05:01 myLuna config file invalid local5 err HTLD[1234]: error : 0 : Cannot get OTT expiry: client 2012 Feb 29 12:05:01 myLuna 192.168.0.100; removing OTT local5 info HTLD[1234]: info : 0 : OTT expiry timer fired for client These messages indicate a problem with the one-time token that is integral to the HTL service. HTL Status 2012 Feb 29 12:05:01 myLuna handler: local5 err HTLD[1234]: error : 0 : Exception in HTL status query This message indicates a failure to get the status of the HTL service. Memory Errors 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Failed to allocate connection request response buffer [size = 100] : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Failed to allocate auth response buffer [size = 100] : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 72 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Failed to allocate key exchange response buffer [size = 100] : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Failed to allocate counter init response buffer [size = 100] : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000001 : Failed to allocate sync beacon response buffer [size = 100] : 192.168.0.100/40847 : RC_MEMORY_ALLOCATION These messages indicate a failure to allocate memory within the HTL service. Client Setup Errors 2012 Feb 29 12:05:01 myLuna local5 err interface : RC_GENERAL_ERROR HTLD[1234]: error : 0xc0000002 : Fail to initialize token 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000002 : Fail to initialize client registration database : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000002 : Fail to populate client registration database : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err HTLD[1234]: error : 0xc0000002 : Fail to reset client registration entry's flags : RC_GENERAL_ERROR 2012 Feb 29 12:05:01 myLuna local5 err manager object : RC_GENERAL_ERROR HTLD[1234]: error : 0xc0000002 : Fail to create token 2012 Feb 29 12:05:01 myLuna Manager : RC_GENERAL_ERROR HTLD[1234]: error : 0xc0000002 : Fail to Initialize VToken local5 err These messages indicate that the HTL service failed to set up the host trust link with an intended client. Connection Errors 2012 Feb 29 12:05:01 myLuna period local5 info HTLD[1234]: info : 0xc0000002: Backup OTT matched in grace 2012 Feb 29 12:05:01 myLuna successfully local5 info HTLD[1234]: info : 0xc0000002: One-time token validated 2012 Feb 29 12:05:01 myLuna token local5 info HTLD[1234]: info : 0xc0000002: Unable to validate one-time 2012 Feb 29 12:05:01 myLuna local5 info secret for dynamic certificate exchange HTLD[1234]: info : 0xc0000002: Unable to derive shared 2012 Feb 29 12:05:01 myLuna certificate and private key local5 info HTLD[1234]: info : 0xc0000002: Unable to create dynamic 2012 Feb 29 12:05:01 myLuna certificate local5 info HTLD[1234]: info : 0xc0000002: Unable to register dynamic 2012 Feb 29 12:05:01 myLuna local5 info and private key sent to client HTLD[1234]: info : 0xc0000002: Sending dynamic certificate 2012 Feb 29 12:05:01 myLuna sent to client HTLD[1234]: info : 0xc0000002: Initial counter information local5 info 2012 Feb 29 12:05:01 myLuna local5 info aligned with increment steps HTLD[1234]: info : 0xc0000002: Received counter is not 2012 Feb 29 12:05:01 myLuna allowable drift range local5 info HTLD[1234]: info : 0xc0000002: Counter is outside 2012 Feb 29 12:05:01 myLuna data; HTL link is up local5 info HTLD[1234]: info : 0xc0000002: Client acknowledged counter Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 73 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna Terminating HTL link. local5 info HTLD[1234]: info : 0xc0000002: HTL message timer expired. 2012 Feb 29 12:05:01 myLuna Entering grace period. local5 info HTLD[1234]: info : 0xc0000002: Client disconnected. 2012 Feb 29 12:05:01 myLuna Cleaning up. local5 info HTLD[1234]: info : 0xc0000002: Client disconnected. 2012 Feb 29 12:05:01 myLuna local5 info connection. Entering grace period. HTLD[1234]: info : 0xc0000002: Server closed HTL 2012 Feb 29 12:05:01 myLuna connection. Cleaning up. HTLD[1234]: info : 0xc0000002: Server closed HTL local5 info These messages indicate a failure in the HTL service’s ability to establish and maintain a connection with a client. Random Number Generator 2012 Feb 29 12:05:01 myLuna local5 crit HTLD[1234]: critical : 0 : Unable to find HSM for RNG This message indicates that the HTL service was unable to connect to the internal HSM for the service’s source of random number data. Network Bus Service Network Bus Service – NBS – is unreleased and undocumented at this time. Remote Backup Service Remote backup service – RBS – is a client-side component of the Luna product line. A future version of Luna Monitoring will describe log messages generated by RBS. Luna Administrative Shell: State and Status The Luna administrator shell – aka lunash (Luna SA) and lush (Luna IS) – is a secure shell for administering the Luna appliance. Facility Keyword syslog or user Software Process lush Log File messages Expected Log Messages The following log messages are normal and expected entries in the log files from various lush commands. Uptime 2012 Feb 29 12:05:01 myLuna syslog info load average: 0.08, 0.05, 0.01 Luna Syslog and SNMP Monitoring Guide lush: uptime before poweroff: 12:05:01 up 4 min, 1 user, 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 74 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna syslog info load average: 0.09, 0.04, 0.01 lush: uptime before reboot: 12:05:01 up 8 min, 1 user, These messages record how long the system had been up before an administrator restarted the Luna appliance via sysconf appliance poweroff or sysconf appliance reboot. Log Cleanup 2012 Feb 29 12:05:01 myLuna syslog info lush: Running the 'syslog cleanup' command, creating tarlogs then deleting all log files except hsm.log ... 2012 Feb 29 12:05:01 myLuna user notice lush: Running the 'syslog cleanup' command; created "logs_cleanup_20120229_0234.tgz" in the scp directory. 2012 Feb 29 12:05:01 myLuna user notice lush: The tar file containing logs is now available via scp as filename "logs_cleanup_20120229_0234.tgz". 2012 Feb 29 12:05:01 myLuna syslog info the log files except hsm.log. lush: Executed the 'syslog cleanup' command. Deleted all 2012 Feb 29 12:05:01 myLuna user notice lush: Executed the 'syslog cleanup' command. Created "logs_cleanup_20120229_0234.tgz" in the scp directory. 2012 Feb 29 12:05:01 myLuna user notice lush: The tar file containing logs is now available via scp as filename "logs_cleanup_20120229_0234.tgz". These messages record that an administrator initiated a log compression and archival operation via syslog cleanup. The first three messages log initiation of the action and the last three messages log completion in the new messages file. Change SSH Port 2012 Feb 29 12:05:01 myLuna user notice Lush: Changing the SSHD listen port number from 22 to 23. This message records that an administrator changed the SSH listening port via sysconf ssh port. Disabled Watchdog Timer 2012 Feb 29 12:05:01 myLuna user info wdt_heartbeat 10: Disabled the WDT heartbeat program. This message records that an administrator disabled the hardware watchdog timer via sysconf appliance watchdog disable. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Failed to Create Log Archive 2012 Feb 29 12:05:01 myLuna syslog info lush: Failed to create logs_cleanup_20120229_0234.tgz: -1. 10 Note that lush executes a utility to disable the watchdog timer. This utility creates the log entry under its process name. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 75 of 102 Luna Monitoring Syslog & SNMP This message indicates that the syslog cleanup command failed to create the backup file: the tar utility returned an error, the last number in the log message. Invalid Signature Mechanism 2012 Feb 29 12:05:01 myLuna user notice lush: ERROR - The current client certificate specifies md5WithRSAEncryption as the signature mechanism. This certificate is no longer useable and must be replaced in order to register the client. Please regenerate the client certificate. This message indicates that the signature of the client certificate found while executing the client register command uses an insecure mechanism. Luna Shell: Command Logging Command logging records administrator actions in a log file. Facility Keyword local5 Software Process lunash [pid] Log File lunalogs Expected Log Messages The following log messages are normal and expected entries in the log files for all lush commands. 2012 Feb 29 12:05:01 myLuna Console local5 info lunash [1234]: info : 0 : Command: hsm show : admin : 2012 Feb 29 12:05:01 myLuna : 192.168.0.100/40847 local5 info lunash [1234]: info : 0 : Command: my file list : monitor Every administrative command executed within the secure Luna shell results in a log message to lunalogs. This message records the command return code (e.g., 0), the command (e.g., hsm show and my file list), the user account (e.g., admin and monitor) and the connection of the secure shell (e.g., Console and 192.168.0.100/40847 – the IP address and outgoing port number on client system). 2012 Feb 29 12:05:01 myLuna Console local5 info lunash [1234]: info : 0 : Lush user login : admin : 2012 Feb 29 12:05:01 myLuna 192.168.0.100/40847 local5 info lunash [1234]: info : 0 : Lush user login : monitor : A slightly different formatted log message records the login action. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please consult the user documentation to seek guidance on how to correct the problem. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 76 of 102 Luna Monitoring Syslog & SNMP Token State and Login Errors 2012 Feb 29 12:05:01 myLuna key local5 warn lunash [1234]: HSM Admin Login: incorrect password or PED 2012 Feb 29 12:05:01 myLuna local5 warn consecutive bad login attempts. lunash [1234]: HSM Admin Login: HSM Zeroized due to three 2012 Feb 29 12:05:01 myLuna password or PED key lunash [1234]: Backup Token Admin Login: incorrect local5 warn 2012 Feb 29 12:05:01 myLuna local5 warn to three consecutive bad login attempts. lunash [1234]: Backup Token Admin Login: HSM Zeroized due On backup and restore operations, these messages indicate failure to successfully authenticate because of either incorrect credentials or the HSM/token being in a state that disallows authentication (i.e., zeroized). Configuration File Error 2012 Feb 29 12:05:01 myLuna local5 info but there is a problem reading it.. lunash [1234]: Config file for overriding hsm slot exists, This message indicates that a lush utility was unable to read the slot number from a configuration file used to define a specific HSM slot identifier. This message is unexpected: contact SafeNet technical support for guidance on what to do next. Node Not Loaded 2012 Feb 29 12:05:01 myLuna local5 err lunash [1234]: Node not loaded: hsm ped set This message indicates that the lush interpreter was unable to load the logged node on start up. This message is unexpected: contact SafeNet technical support for guidance on what to do next. cron & crontab The Luna appliance relies upon the cron process to perform regular operations. Facility Keyword syslog Software Process crontab Log File messages Facility Keyword cron Software Process anacron[pid] Log File cron-yyyy-mm-dd Facility Keyword auth Software Process crond(pam_unix)[pid] Log File secure Expected Log Messages The following log messages are normal and expected entries in the log files from cron activities. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 77 of 102 Luna Monitoring Syslog & SNMP Updated Timestamp 2012 Feb 29 04:02:01 myLuna to 2012-02-29 cron notice anacron[1234]: Updated timestamp for job `cron.monthly' 11 This message indicates that the cron process ran the scheduled monthly (or daily or weekly) cron job. Login and Logout 2012 Feb 29 12:05:01 myLuna (uid=0) auth info crond(pam_unix)[1234]: session opened for user root by 2012 Feb 29 12:05:01 myLuna auth info crond(pam_unix)[1234]: session closed for user root These messages record that cron opened a session to complete any scheduled activities and closed the session when done. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Disk Filling 75% - 89% 2012 Feb 29 12:00:00 myLuna syslog warn crontab: DISK FULL: WARNING! First log threshold reached 75%. Please clean up the logs using the "syslog cleanup" command 90% - 94% 2012 Feb 29 12:00:00 myLuna deleted. syslog alert crontab: DISK FULL: 90%. All log files except hsm.log 2012 Feb 29 12:00:00 myLuna syslog alert crontab: DISK FULL: Second log threshold reached 90%. Created logs_diskFull_20120229_1234.tgz and deleted all log files. 12 2012 Feb 29 12:00:00 myLuna syslog alert 90% policy: tarlogs_cleanup. 13 crontab: DISK FULL: WARNING! Second log threshold reached 2012 Feb 29 12:00:00 myLuna syslog alert crontab: DISK FULL: Second log threshold reached 90%. Created logs_diskFull_20120229_1234.tgz and deleted all log files. 14 2012 Feb 29 12:00:00 myLuna syslog alert crontab: DISK FULL: The tar file containing logs is now available via scp as filename "logs_diskFull_20120229_1234.tgz 15 2012 Feb 29 12:00:00 myLuna syslog alert crontab: DISK FULL: Second log threshold reached $percent%. Failed to create "logs_diskFull_20120229_1234.tgz 16 11 Or `cron.daily' or `cron.weekly'. Message written before syslog rotates logs. 13 Message written before syslog rotates logs. 14 Message written after syslog rotates logs. 15 Message written before and after syslog rotates logs. 16 Message written if tar command failed to create file. 12 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 78 of 102 Luna Monitoring Syslog & SNMP 95% & > 2012 Feb 29 12:00:00 myLuna syslog emerg crontab: DISK FULL: EMERGENCY! Third log threshold reached 95%. Please clean up the logs using the "syslog cleanup" command 2012 Feb 29 12:00:00 myLuna reached 95%. Stopping NTLS syslog emerg 2012 Feb 29 12:00:15 myLuna syslog emerg reached 95%. Stopping syslog soon crontab: DISK FULL: EMERGENCY! Third log threshold crontab: DISK FULL: EMERGENCY! Third log threshold These messages indicate that disk use is reaching thresholds that require intervention to remove files. Read-Only File System 2012 Feb 29 04:02:01 myLuna cron err anacron[1234]: Cannot exclusively create /var/run/anacron.pid: Read-only file system 2012 Feb 29 04:02:01 myLuna cron err running.: Resource deadlock avoided anacron[1234]: Cannot run in daemon mode - anacron already 2012 Feb 29 04:02:01 myLuna anacron[1234]: Aborted cron err These messages indicate that the file system is read-only. A possible root cause is a Linux journaling error or a failing hard drive. Certificate Monitoring Daemon The certificate monitoring daemon watches for an impending expiry of the NTLS certificate and sends a trap when the lifetime of the certificate falls within a configurable threshold number of days remaining. Facility Keyword local5 Software Process certmonitord[pid] Log File lunalogs Expected Log Messages The following log messages are normal and expected entries in the log files when NTLS certificate monitoring is enabled. Daemon Started 2012 Feb 29 12:05:01 myLuna monitor started local5 info certmonitord[1234]: info : 0 : NTLS certificate expiry 2012 Feb 29 12:05:01 myLuna local5 info certmonitord[1234]: info : 0 : NTLS certificate expiry monitor is configured to send SNMP trap 5 day(s) before the NTLS certificate expires and on every 12 hour(s) These messages indicate that the certificate monitoring daemon is running. The daemon does not run by default. Rather, an administrator must configure and start it from the Luna administrative shell. The number of days and hours in the message reflects the configuration set via Lush. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 79 of 102 Luna Monitoring Syslog & SNMP Daemon Stopping 2012 Feb 29 12:05:01 myLuna local5 info certificate expiry monitor.... certmonitord[1234]: info : 0 : Shutting down NTLS 2012 Feb 29 12:05:01 myLuna monitor terminated certmonitord[1234]: info : 0 : NTLS certificate expiry local5 info These messages indicate that the certificate monitoring daemon gracefully shut down as a result of a signal (SIGINT, SIGTERM, SIGABRT) outside of a normal system shutdown (e.g., Lush ntls certificate monitor disable). Impending Certificate Expiry 2012 Feb 29 12:05:01 myLuna local5 info expire on Jul 26 16:32:48 2023 GMT certmonitord[1234]: info : 0 : NTLS certificate will be 2012 Feb 29 12:05:01 myLuna local5 info certmonitord[1234]: info : 0 : NTLS certificate expiry SNMP trap sent to trap host 192.168.0.115 These messages indicate that the NTLS certificate is set to expire and that the certificate monitoring daemon successfully sent a trap to the configured host. Certificate Missing 2012 Feb 29 12:05:01 myLuna missing local5 warn certmonitord[1234]: warning : 0 : NTLS certificate is This message indicates that the daemon failed to find the server.pem file for NTLS in the expected location on the hard drive. However, the daemon remains running in the event that an administrator creates the necessary server certificate in a subsequent operation. On a new Luna appliance from the factory, this message is normal. An administrator must create the NTLS certificate (sysconf regenCert). New NTLS Certificate 2012 Feb 29 12:05:01 myLuna local5 info certmonitord[1234]: info : 0 : New NTLS certificate detected and the expiry date of this new certificate is Jul 26 16:32:48 2033 GMT This message indicates that an administrator created a new NTLS certificate that is sufficiently far into the future such that a trap is no longer necessary. The daemon will continue to monitor for the certificate expiry window. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Failed to Detach 2012 Feb 29 12:05:01 myLuna console local5 err Luna Syslog and SNMP Monitoring Guide certmonitord[1234]: error : 0 : Failed to detach from 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 80 of 102 Luna Monitoring Syslog & SNMP This message indicates that the startup procedure for the certificate monitoring daemon failed, specifically that the daemon did not launch into a background process. Running in Console Mode 2012 Feb 29 12:05:01 myLuna local5 info monitor running in console mode certmonitord[1234]: info : 0 : NTLS certificate expiry This message indicates that the certificate monitoring daemon is running in console mode rather than as a background process. SNMP V3 Not Properly Configured 2012 Feb 29 12:05:01 myLuna properly configured local5 info certmonitord[1234]: info : 0 : SNMP v3 trap is not This message indicates that either the engine identifier and/or the host IP address configured and stored in the snmp.conf is/are invalid. Lush command(s) that create these entries include the necessary processing checks to ensure the operation(s) writes valid entries to the configuration file. Failed to Allocate Memory Buffers 2012 Feb 29 12:05:01 myLuna buffers local5 err certmonitord[1234]: error : 0 : Failed to allocate memory This message indicates that the daemon was unable to allocate the requisite buffers for file handling and string manipulation. Failed to Send Trap 2012 Feb 29 12:05:01 myLuna local5 err certmonitord[1234]: error : 0 : Failed to send NTLS certificate expiry SNMP trap to trap host 192.168.0.100 This message indicates that the certificate monitoring daemon was unable to execute a system call with a pre-formed command to send a trap. The daemon relies upon the Linux utility snmptrap() to complete this action. An invalid host IP address for example, would cause the system call to fail (e.g., 192.168.0.1004). certmonitord Crash and Burn 2012 Feb 29 12:05:01 myLuna local5 crit certmonitord[1234]: info : 0 : certmonitord CRASH AND BURN! Stack dump saved to /var/log/certmonitord_bt_2012-02-29_12:05:01 2012 Feb 29 12:05:01 myLuna local5 crit and unable to dump the stack! certmonitord[1234]: info : 0 : certmonitord CRASH AND BURN These messages indicate a programming error. The first message indicates that the certificate monitoring daemon terminated abnormally (on one of SIGSEGV, SIGILL or SIGBUS signals), generating a stack trace file certmonitord_bt_2012-02-29_12:05:01 in the process. Forwarding this file to SafeNet product Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 81 of 102 Luna Monitoring Syslog & SNMP engineering may assist a developer to isolate the reason for the abnormal termination. The second message indicates an abnormal termination but with no resulting stack trace created. Luna SNMP Trap Agent Luna SNMP Trap Agent or lsta looks for significant events within the Luna appliance. When it sees a worthy event, lsta formats a trap notification and sends it to the configured user. Facility Keyword local5 Software Process lsta Log File lunalogs Expected Log Messages The following log messages are normal and expected entries in the log files when lsta monitoring is enabled and started. Agent Starting 2012 Feb 29 12:05:01 myLuna local5 info lsta: Starting Luna SNMP Trap Agent This message indicates that lsta is starting. The agent does not run by default. Rather, an administrator must configure and start it from the Luna administrative shell. Creating Named Pipe 2012 Feb 29 12:05:01 myLuna local5 info lsta: Created /home/admin/syslog.pipe as a named pipe This message indicates that lsta created the necessary named pipe for interprocess communication with the syslog facility. This message results the first time lsta is started on a new Luna appliance. Termination 2012 Feb 29 12:05:01 myLuna call local5 info lsta: Luna SNMP Trap Agent terminated: Interrupted system This message indicates that lsta caught a SIGINT, SIGABRT or SIGTERM signal and terminated. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. SNMP Data Logged 2012 Feb 29 12:05:01 myLuna local5 info lsta: fanAttentionNotify for [myLUT:192.168.0.58 / messages / 2012 Feb 29 12:05:00 / ipmievd / 1] When trace level debugging is enabled, this message records the SNMP trap notification data sent by lsta. Trace level debugging is off in Luna SA so this message is unexpected. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 82 of 102 Luna Monitoring Syslog & SNMP Missing SNMP Parameters 2012 Feb 29 12:05:01 myLuna local5 err lsta: Error opening /usr/local/share/snmp/snmp.conf This message indicates that the SNMP parameters needed by lsta are missing. A likely root cause is that an administrator has not yet set the parameters via the sysconf snmp trap set command. snmptrap Failed 2012 Feb 29 12:05:01 myLuna local5 err lsta: snmptrap failed: 256 This message indicates that lsta was unable to send a formatted trap notification to the intended user. A possible root cause is an out-of-date SAFENET-APPLIANCE-MIB on the appliance or a parameter defined by the sysconf snmp trap set command in error (e.g., invalid IP address). Interprocess Communication Failures 2012 Feb 29 12:05:01 myLuna local5 err lsta: Pipe file disappeared, stopping lsta 2012 Feb 29 12:05:01 myLuna local5 err lsta: Cannot read from pipe /home/admin/syslog.pipe: 5 2012 Feb 29 12:05:01 myLuna local5 err lsta: Will not overwrite /home/admin/syslog.pipe 2012 Feb 29 12:05:01 myLuna local5 err lsta: Cannot mknod /home/admin/syslog.pipe: 13 These messages indicate failure with the named pipe used for interprocess communication with the syslog facility. The first two messages represent failures during lsta operation; the last two on start up. Unknown Sub-system 2012 Feb 29 12:05:01 myLuna local5 err lsta: Unknown/unsupported sub-system: xxx This message indicates a programming error. lsta should have logic for all sub-systems defined in the SAFENET-APPLIANCE-MIB. Log Record Parsing 2012 Feb 29 12:05:01 myLuna local5 info lsta: Could not parse log record: This message indicates that lsta could not properly parse a log record for the fields expected. is the log message received by lsta from rsyslogd. Could Not Close Parameters File 2012 Feb 29 12:05:01 myLuna local5 err lsta: Error closing /usr/local/share/snmp/snmp.conf This message indicates that lsta could not properly close the SNMP parameters file. Could Not Close Pipe 2012 Feb 29 12:05:01 myLuna local5 err lsta: Error closing /home/admin/syslog.pipe This message indicates that lsta could not properly close the named pipe used by rsyslog to communicate log records. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 83 of 102 Luna Monitoring Syslog & SNMP chdir Failed 2012 Feb 29 12:05:01 myLuna local5 err lsta: Could not chdir This message indicates that lsta could not properly change working directories. umask Failed 2012 Feb 29 12:05:01 myLuna local5 err lsta: Could not umask This message indicates that lsta could not properly determine the umask file settings. Could Not Open stdin 2012 Feb 29 12:05:01 myLuna local5 err lsta: Error opening stdin This message indicates that lsta could not properly open standard input. Could Not Open stderr 2012 Feb 29 12:05:01 myLuna local5 err lsta: Error opening stderr This message indicates that lsta could not properly open standard error. Admin API Admin API is the process that supports administration for the appliance via REST. Facility Keyword local5 Software Process AdminAPI Log File lunalogs Expected Log Messages The following log messages are normal and expected entries in the log files when the webserver for the REST API is running. Web Server Started 2012 Feb 29 12:05:01 myLuna 192.168.0.79:8443 local5 info AdminAPI[1234]: info : 0 : Server is setting up on 2012 Feb 29 12:05:01 myLuna local5 info AdminAPI[1234]: info : 0 : Server cipher set: 2012 Feb 29 12:05:01 myLuna local5 info AdminAPI[1234]: info : 0 : Server is using TLS1.2 only 2012 Feb 29 12:05:01 myLuna ssl_key.pem local5 info AdminAPI[1234]: info : 0 : Server is loading key file: 2012 Feb 29 12:05:01 myLuna ssl_cert.pem local5 info AdminAPI[1234]: info : 0 : Server is loading cert file: 2012 Feb 29 12:05:01 myLuna dh2048.pem local5 info AdminAPI[1234]: info : 0 : Server is loading dh file 2012 Feb 29 12:05:01 myLuna local5 info NID_X9_62_prime256v1 named curve AdminAPI[1234]: info : 0 : Server is using 2012 Feb 29 12:05:01 myLuna AdminAPI[1234]: info : 0 : Server finished setting up local5 info Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 84 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna hmac_sha256 local5 info AdminAPI[1234]: info : 0 : Request hashing set: 2012 Feb 29 12:05:01 myLuna rsa_pkcs1 local5 info AdminAPI[1234]: info : 0 : Asymmetric encryption set: 2012 Feb 29 12:05:01 myLuna aes_256_cfb local5 info AdminAPI[1234]: info : 0 : Symmetric encryption set: These messages indicate that web server started successfully. The web server does not run by default. Rather, an administrator must configure and start it from the Luna administrative shell. Client Connected 2012 Feb 29 12:05:01 myLuna connected local5 info AdminAPI[1234]: info : 0 : Client 192.168.0.120:52500 2012 Feb 29 12:05:01 myLuna request local5 info AdminAPI[1234]: info : 0 : Parsed client 192.168.0.120's 2012 Feb 29 12:05:01 myLuna local5 info AdminAPI[1234]: info : 0 : Client 192.168.0.120SESSION_ID=413ef8fe-3667-4083-ae2f-44e66e1ac70e requested POST:/lunasa/services/ntls/clients 2012 Feb 29 12:05:01 myLuna local5 info AdminAPI[1234]: info : 0 : 192.168.0.120 successfully logged in as admin with session id of 413ef8fe-3667-4083-ae2f-44e66e1ac70e These messages indicate that a client connected without issue to the web server and that the web server was able to fulfill the client’s request. Unexpected Log Messages Under normal circumstances, you should not see any of these log messages. If you do so, please contact SafeNet technical support to report the message and seek guidance on what to do next. Web Server Busy 2012 Feb 29 12:05:01 myLuna local5 err refused because server is busy AdminAPI[1234]: error : 0 : Client 192.168.0.120 connection This message indicates that the queue of client connections is full and the web server is unable to respond to any additional request. Client Session Problems 2012 Feb 29 12:05:01 myLuna local5 err a request without a session token AdminAPI[1234]: error : 0 : Client 192.168.0.120 submitted 2012 Feb 29 12:05:01 myLuna a bad cookie local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120 submitted 2012 Feb 29 12:05:01 myLuna session local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120 has no 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 session keys do not match an active session 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 requested access to GET:/api/lunasa/hsms/4343343 and access was denied 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 failed to decrypt packet Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 85 of 102 Luna Monitoring Syslog & SNMP 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 requested a non-existent resource 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 sent a bad payload 2012 Feb 29 12:05:01 myLuna local5 err AdminAPI[1234]: error : 0 : Client 192.168.0.120-7e33c7bb54a0-4a92-a031-4e08fc90cfa5 failed to include authentication token 2012 Feb 29 12:05:01 myLuna local5 err failed to establish SSL connection AdminAPI[1234]: error : 0 : Client 192.168.0.120:52500 2012 Feb 29 12:05:01 myLuna failed local5 err AdminAPI[1234]: error : 0 : 192.168.0.120 login attempt 2012 Feb 29 12:05:01 myLuna a031-4e08fc90cfa5 timed out local5 err AdminAPI[1234]: error : 0 : Session 7e33c7bb-54a0-4a92- 2012 Feb 29 12:05:01 myLuna local5 err a031-4e08fc90cfa5 logged out AdminAPI[1234]: error : 0 : Session 7e33c7bb-54a0-4a92- These messages indicate problems with the client session. Unknown Resource 2012 Feb 29 12:05:01 myLuna non-existent resource local5 err AdminAPI[1234]: error : 0 : Internal action requested a This message indicates that the REST framework could not find the resource requested. Blacklisted Clients 2012 Feb 29 12:05:01 myLuna blacklist local5 warn 2012 Feb 29 12:05:01 myLuna local5 crit 192.168.0.120 attempted to connect AdminAPI[1234]: warning : 0 : Added 192.168.0.120 to the AdminAPI[1234]: critical : 0 : Blacklisted user These messages indicates that the REST server has blacklisted a client. The first indicates the action of adding a client to the black list; the second indicates a subsequent attempt by a blacklisted client to connect. Other Appliance Logging hsm information show Output from the Lush hsm information show command includes additional log data as shown in the following example. [local_host] lunash:>hsm information show HSM Event Counters: Operation Requests: Operation Errors: Crypto Operation Requests: Crypto Operation Errors: Critical Events: Luna Syslog and SNMP Monitoring Guide 21998 1707 0 0 0 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 86 of 102 Luna Monitoring Syslog & SNMP Non-Critical Events: 34 sysstatd, snmp and ntls periodically make calls to the HSM and these calls result in incremented counters for Operation Requests and Operation Errors. luna-snmp polls the HSM every minute and increments these two counters by 49 and 1 respectively. sysstatd polls the HSM with each rotation of the LCD messages (~15 to 20 seconds) and increments these two counters by 25 and 3 respectively. ntls polls the HSM repeatedly and increments the Operation Requests counter by more than 600 per minute. Crypto Operation Requests and Crypto Operation Errors reflect counts from client application requests to the HSM. Critical Events are catastrophic failure of the HSM firmware as reported by the HSM itself. An increment in this counter also results in a crash log file and a syslog message (messages) similar to the following example: 2012 Feb 22 03:15:13 LunaSA5 local6 crit oamp[2318]: CRIT: Fatal error 0x00300203 2012 Feb 22 03:15:13 LunaSA5 local6 info oamp[2318]: info : 0 : HSM critical event detected. port dump generated : hsm_dump_20120222031513 Dual Non-Critical Events are either ERR or INFO messages logged to hsm.log. A count the INFO and ERR messages in the following excerpt reflects the 34 reported in the example above. 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: ----- HSM Logging started 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: Warm boot 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: FW Rev 6.2.1-10 2014 Aug 21 07:18:14 local_host SOURCE/LUNA2/MAIN_MOD/main.c local6 info oamp[2100]: INFO: File: 2014 Aug 21 07:18:14 local_host 16:53:13 local6 info oamp[2100]: INFO: Date: Jul 29 2011, Time: 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: POWER-UP LOG DUMP START 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: Power-up log created 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: Appending power-up log... 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: HSM is powered-up 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: (YYYY:MM:DD:hh:mm:ss = 0000:01:01:00:00:05.04) 2014 Aug 21 07:18:14 local_host HSM time: 0 min local6 err oamp[2100]: ERR: RTC: lost all power 2014 Aug 21 07:18:14 local_host local6 err oamp[2100]: ERR: RTC: zeroized on power loss 2014 Aug 21 07:18:14 local_host Clock restarted. local6 info oamp[2100]: INFO: 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: decommission...LOG(INFO): POWER-UP LOG DUMP END 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: (YYYY:MM:DD:hh:mm:ss = 0000:01:01:00:00:53.21) Luna Syslog and SNMP Monitoring Guide RTC: re-configuring...passed. Zeroizing HSM after Erasing power-up log HSM time: 0 min 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 87 of 102 Luna Monitoring Syslog & SNMP 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: MM_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: PM_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: I2C_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: RTC_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: NVRAM_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: 0x00000000 KeyStatus: 0x10000000 0x0000006D ES Flags: 0x00000002 2014 Aug 21 07:18:14 local_host local6 info 2014 Aug 21 07:18:14 local_host for unknown reason local6 err oamp[2100]: INFO: oamp[2100]: ERR: CGX Ver: 0xC0020802 HA Mode: CGX_Init OK MTK: security function was zeroized 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: MTK_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: sxl_init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: PE1746_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: RN_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: OH_Init OK 2014 Aug 21 07:18:14 local_host local6 info disabled, skipping all self tests oamp[2100]: INFO: CA_AlgorithmSelfTest(): HW 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: CA_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: UM_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: SM_Init OK 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: Supported callback I/O v.1 2014 Aug 21 07:18:14 local_host local6 info oamp[2100]: INFO: Supported callback protocol v.1 Client Logging To this point, the logs described apply to the Luna appliance. The Luna client also generates log files. This section describes client-side logging, specifically, logs generated by the following facilities: • • cklog HA log. cklog A future version of Luna Monitoring will identify and describe these messages. HA log A future version of Luna Monitoring will identify and describe these messages. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 88 of 102 Luna Monitoring Syslog & SNMP SNMP Introduction Simple Network Management Protocol provides a procedure to collect information from devices in a network. Luna appliances support SNMP through queries (e.g., “walking” managed information base trees) and through asynchronous notification of events (i.e., traps). This version of Luna Monitoring describes the support for traps in Luna appliances. Traps For the purpose of making the examples that follow easier to describe, assume that your name is Pete. While it may not be evident just yet, the subject of traps and the preceding text that describes syslog messages are related. On the appliance is the Luna SNMP Trap Agent or lsta. When you configure and enable traps, lsta runs as a background process and uses the named pipe feature of the system logging facility (rsyslog) to receive messages directed to log files. Thus, lsta receives a copy of all log messages. Using some of the rules for interpreting log messages described in the “Interpreting Logs” section above, lsta generates traps for significant events. Luna SA uses a version of rsyslogd that supports logging output to named pipes (fifos). While the man pages for rsyslog states that "this is handy for debugging," nothing restricts the feature for other uses and SafeNet has leveraged it for SNMP trap generation. lsta creates the fifo the first time the process starts up. The section below, “Luna Appliance Trap Events” describes the significant events for which the Luna appliance generates traps. Configuring and Enabling Traps on Luna Appliance The following list identifies the steps to configure and enable traps on a Luna appliance. Each step corresponds to an administrative command via the Luna shell. 1. Add an SNMP user to the system: sysconf snmp user add. 2. Set the SNMP trap parameters for the user created in the preceding step: sysconf snmp trap set. 3. Enable and disable SNMP traps: sysconf snmp traps enable/disable. 4. Restart the system logging service: service restart syslog –force. 5. Enable SNMP traps once again. Refer to the reference material for details of the parameters to each of the Lush commands to perform these steps. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 89 of 102 Luna Monitoring Syslog & SNMP Assuming that you use Pete as the snmp user, here are text strings that you can cut-and-paste directly into a Lush session to configure and enable traps. You need to make one adjustment: the IP address for sysconf snmp trap set must match the target for SNMP notifications. The examples use 192.168.0.100. sysconf snmp user add -secName pete -authPassword PASSWORD -authProtocol SHA -privPassword PASSWORD -privProtocol AES sysconf snmp trap set -h 192.168.0.100 -secName pete -e 1234567890 authpr SHA -authpw PASSWORD -privPr AES -privPw PASSWORD sysconf snmp trap enable sysconf snmp trap disable service restart syslog -force sysconf snmp trap enable Configuring Trap Notification Once you have traps configured and enabled on a Luna appliance, you probably want to test that you can successfully receive a trap notification. The description that follows explains how to configure a CentOS Linux virtual machine to test trap notifications. Although your test environment is likely different, these instructions should provide enough information to assist you in your efforts. Install net-snmp You need an SNMP trap daemon to receive trap notifications if you follow the same test process used herein. Use yum to install net-snmp: yum install net-snmp If you intend to do development, you should also install: yum install net-snmp-utils yum install net-snmp-devel Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 90 of 102 Luna Monitoring Syslog & SNMP Create a Trap Handler When you receive a trap notification, you should do something with it. Consider the shell script example from the net-snmp tutorial to create a trap handler. Here is the URL for the tutorial: http://www.net-snmp.org/tutorial/tutorial-5/commands/snmptrap.html And here is an example shell script trap handler. You can cut-and-paste this text into a file that you save as /etc/snmp/traps to align with subsequent instructions. #!/bin/sh read host read ip vars= count=1 while read oid val do count=$[count+1] if [ "$vars" = "" ] then vars="$oid = $val" else vars="$vars, $oid = $val" fi done echo a $1 trap from host=$host at IP $ip vars=$vars>>/var/log/traplogs Configure snmptrapd.conf Navigate to /etc/snmp and you should find a file there: snmptrapd.conf. Use your favorite editor to modify this file. Cut and paste in the following lines after the commented out traphandle. Note that the example assumes that you are using “pete” for the SNMP user and that it only supports three of the SNMP trap types of lsta. traphandle SAFENET-APPLIANCE-MIB::fanAttentionNotify /etc/snmp/traps Fan traphandle SAFENET-APPLIANCE-MIB::powerSupplyAttentionNotify /etc/snmp/traps PSU traphandle SAFENET-APPLIANCE-MIB::motherboardAttentionNotify /etc/snmp/traps Motherboard ########################################################################### # SECTION: Runtime options Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 91 of 102 Luna Monitoring Syslog & SNMP # # Runtime options disableAuthorization no createUser -e 1234567890 pete SHA "PASSWORD" AES "PASSWORD" authUser log,execute,net pete Add the SafeNet MIBs Navigate to /usr/share/snmp/mibs and securely copy (scp) the following MIBs to this directory: CHRYSALIS-UTSP-MIB.txt SAFENET-APPLIANCE-MIB.txt SAFENET-GLOBAL-MIB.txt SAFENET-HSM-MIB.txt SNMPv2-SMI.txt Enable Trap Packets Through the Firewall On your Linux virtual machine, iptables may block SNMP trap packets by default. Perform the following steps to overcome this restriction. Stop iptables: /etc/init.d/iptables stop Edit /etc/sysconfig/iptables and add the following two lines before the first REJECT directive in the file: ######## Allow SNMP trap packets. -A INPUT -p udp --dport 162 -j ACCEPT You do not need to include the comment. For reference, the first REJECT directive in the file on the example VM is: -A INPUT -j REJECT --reject-with icmp-host-prohibited Note that the order of specification is important. Start iptables: /etc/init.d/iptables start Disable SELinux SELinux may be enabled on your Linux virtual machine. You need to disable SELinux to start the snmptrapd in a subsequent step. To disable SELinux, type the following at a terminal prompt: echo 0 >/selinux/enforce Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 92 of 102 Luna Monitoring Syslog & SNMP Start the SNMP Trap Daemon Start the SNMP trap daemon in debug mode to see the packets received: snmptrapd -Dusm -d -f -Le. Generate a trap on the appliance – details of how to do that shortly – and receive the trap packet and a message logged to /var/log/traplogs. Stop the daemon and start it again, this time as a service: service snmptrapd start. Generate a trap on the appliance and note a message written to /var/log/traplogs as an indication of a successful trap notification. NOTE: If you choose not to start snmptrapd in debug mode, you must start snmptrapd twice (i.e., start, stop, start). This first invocation constructs the snmptrapd.conf file on the appliance; the second invocation opens and uses this configuration file. NOTE: Either the snmptrap command on the appliance or snmptrapd appears to compress multiple spaces to a single space. If you attempt to do exact pattern matching in a script, keep this point in mind. By way of an example, the following appliance message: 2014 Apr 3 15:47:30 myLUT . Lower Critical going low daemon notice ipmievd: ***TEST : SEQNO_10000 : Fan sensor Fan1A (Reading 2000 .lt Threshold 2000 RPM) results in a traplog message: a Fan trap from host= at IP UDP: [xxx.xxx.xxx.xxxx]:47478->[xxx.xxx.xxx.xxx]:162 vars=system.sysUpTime.sysUpTimeInstance = 0:0:28:12.33, .iso.org.dod.internet.snmpV2.snmpModules.snmpMIB.snmpMIBObjects.snmpTrap.snmpTrapOID.0 = enterprises.safenet-inc.safenetRoot.luna.appliance.ssTraps.fanAttentionNotify, enterprises.safenetinc.safenetRoot.luna.appliance.ssTraps.ssLogReference = [myLUT:xxx.xxx.xxx.xxx / messages / 2014 Apr 3 15:47:30 / ipmievd / 1] Only a single space separates “Apr” and ‘3’ in the latter message while two spaces do so in the former message. Testing Trap Events on Luna Appliance In all likelihood, your Luna appliance is in a rack in a secure room somewhere relatively far away from where you sit. If locally available, you can generate traps by disconnecting power to one of the two supplies. Short of this action, you have no way to initiate the events that result in traps. That leaves you with little to develop and test a monitoring facility for SNMP traps. To test that you have configured everything successfully to generate and receive trap notifications, SafeNet added a Lush command for this purpose: sysconf snmp trap test. The command takes several parameters – refer to the reference material for details of the parameters Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 93 of 102 Luna Monitoring Syslog & SNMP NOTE: This command writes a test message to the applicable system log file. The command neither checks nor reports the status of lsta. If lsta is not running, the appliance does not generate a trap. Before running the command sysconf snmp trap test, ensure that you have started lsta using the sysconf snmp trap enable command. Using the Examples In the section that follows, the text includes a fully-formed Lush command that you can copy and paste into a Lush session to initiate a trap notification. Use any one of these examples to test that you have successfully configured the Luna appliance and your trap handler. NOTE for Mozilla Firefox Users: Firefox 19 introduced its own integrated PDF viewer, which contains a bug that adds extra line feeds to any text that you copy and paste from the viewer. Any examples you attempt to copy and paste into Lush from the integrated PDF viewer will fail. To fix this issue, you need to configure Firefox to use the Acrobat plug-in instead of the integrated viewer: 1. Open the Firefox menu and click on Options to display the Options dialog. 2. Click on the Applications tab in the Options dialog. 3. Change the action for the Portable Document Format (PDF) from Preview in Firefox to either Use Acrobat Reader (in Firefox) or Use Adobe Reader. Luna Appliance Trap Events To avoid covert channel attacks via SNMP traps, the Luna appliance generates categories of traps. Contained within the trap notification is sufficient information for an administrator to query the Luna appliance for the specific event that led to the trap. Appliance MIB Overview The SAFENET-APPLIANCE-MIB.txt file defines the management information base for Luna appliance traps. Here is a segment of the MIB: ssTraps OBJECT IDENTIFIER ::= { appliance 2 } ssLogReference OBJECT-TYPE SYNTAX DisplayString (SIZE (0..256)) MAX-ACCESS read-only STATUS current DESCRIPTION "This object indicates the log record pertaining to the sub-system trap." ::= { ssTraps 1 } Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 94 of 102 Luna Monitoring Syslog & SNMP fanAttentionNotify NOTIFICATION-TYPE OBJECTS { ssLogReference } STATUS current DESCRIPTION "Notify that a fan needs attention." ::= { ssTraps 2} fanAttentionNotify is the notification type for a fan-related event. Corresponding AttentionNotify notification types define the other sub-systems for which a Luna appliance reports significant events. The relevant information reported by a notification is the ssLogReference object. Specifically, this object provides: the hostname and IP address of the appliance; the log file that contains the event; the date and time of the event; the process that reported the event in the log; and a boolean value that indicates whether the event is an assert (1) or de-assert condition. With these data, an administrator can login to the Luna appliance and query the appropriate log file for the details of the trap event. Sub-System Log Reference An example might help to explain the sub-system log reference object of the appliance MIB. Assume a power supply fails. Via the intelligent platform monitoring interface, the ipmievd process learns of this failure and generates a log message. ipmievd sends the message to the rsyslogd process. In addition to writing a record to the messages log file, rsyslogd writes the record to the named pipe at which lsta is listening. lsta determines that this event is trap-worthy. The agent formats the necessary parameters and uses the net-snmp snmptrap command to send a notification for a SAFENETAPPLIANCE-MIB::powerSupplyAttentionNotify notification with a SAFENET-APPLIANCEMIB::ssLogReference object to the configured SNMP V3 user. The trap handler receives the notification in a packet that includes the following example segment: SAFENET-APPLIANCE-MIB::powerSupplyAttentionNotify, SAFENET-APPLIANCE-MIB::ssLogReference = [myLuna:192.168.0.101 / 2012 Feb 29 12:05:01 / messages / ipmievd[1234] / 1] where • • • • • • myLuna is the hostname of the Luna appliance 192.168.0.101 is the IP address of the first Ethernet interface on the appliance messages is the log file that contains the event leading to the trap notification 2012 Feb 29 12:05:01 is the date and timestamp recorded in the log file ipmievd[1234] is the process that logged the message. 1 is a boolean that indicates whether the trap is for an assertion (1) or de-assertion (0) event. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 95 of 102 Luna Monitoring Syslog & SNMP This information gives you what is needed to identify the specific log entry that led to the trap. If you log into the appliance and look at the messages log file: [myLuna] lunash:>syslog tail -logname messages you see the following entry: 2012 Feb 29 12:05:01 myLuna local4 notice ipmievd[1234]: ***TEST : Power Supply sensor PSU2_Status . - Failure detected Asserted From this log message, you know that the second power supply unit has failed and you can dispatch a technician to investigate. Note that the Luna appliance tags log messages generated by the sysconf snmp trap test command with a ***TEST designator. This designator allows you to determine legitimate events from test events. Fan Failure lsta generates a trap for a fan failure based on the following conditions for the ipmievd process and with any of { “Fan1A”, “Fan1B”, “Fan2A”, “Fan2B”, “Fan3A”, “Fan3B”} in the body of the message: • • Body of log message contains the text “'Lower Critical going low” and the threshold reported represents an assert condition Body of log message contains the text “'Lower Non-recoverable going low” and the threshold reported represents an assert condition. Recall from “Threshold Events” in this document that IPMI reports assert and de-assert conditions. A “true” relationship to the “Reading xxxx < Threshold yyyy RPM)” segment of the log message represents an assert event. A “false” relationship represents a de-assert event. Fan failures correspond to the fanAttentionNotify NOTIFICATION-TYPE of the SAFENETAPPLIANCE-MIB. Note that the Luna administrative shell prohibits the ‘<’ and ‘>’ characters as parameters. But some traps key off threshold readings that rely on this arithmetic comparator. To enable test log messages of this sort, use a “.lt” or “.gt” string in place of the ‘<’ or ‘>’ character in the formatted Lush command. You can cut-and-paste the following examples in a Lush session to create test log messages that generate fan traps (the first, second, fourth and fifth examples create assert events; the third and sixth examples, deassert events) : Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 96 of 102 Luna Monitoring Syslog & SNMP sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Critical going low (Reading 0 .lt Threshold 2000 RPM)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Critical going low (Reading 2000 .lt Threshold 2000 RPM)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Critical going low (Reading 21000 .lt Threshold 2000 RPM)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Non-recoverable going low (Reading 500 .lt Threshold 1000 RPM)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Non-recoverable going low (Reading 1000 .lt Threshold 1000 RPM)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Fan sensor Fan3B . Lower Non-recoverable going low (Reading 5100 .lt Threshold 1000 RPM)" Power Supply Failure lsta generates a trap for a power supply failure based on the following conditions for the ipmievd process and with any of {” PSU1_Status”, "PSU1_+12V_value", "PSU1 Temp_value", "PSU1 FAN_value", “PSU2_Status”, "PSU2_+12V_value", "PSU2 Temp_value", "PSU2 FAN_value", “Power Supply”} in the body of the message: • • • • • • • Body of log message contains the text “Failure detected Asserted” Body of log message contains the text “Failure detected Deasserted” Body of log message contains the text “Presence detected Asserted” Body of log message contains the text “Presence detected Deasserted” Body of log message contains the text “- Transition to Power Off” Body of log message contains the text “'Lower Non-recoverable going low” and the threshold reported represents an assert condition Body of log message contains the text “'Upper Non-recoverable going high” and the threshold reported represents an assert condition. Power supply failures correspond to the powerSupplyAttentionNotify NOTIFICATION-TYPE of the SAFENET-APPLIANCE-MIB. Here is text you can use to create power supply traps: sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Power Supply sensor PSU2_Status . - Failure detected Asserted" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Power Supply sensor PSU1_Status . - Presence detected Deasserted" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Power Supply sensor - Transition to Power Off" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor PSU2_+12V_value. Upper Non-recoverable going high (Reading 14.538 .gt Threshold 13.392 Volts)" Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 97 of 102 Luna Monitoring Syslog & SNMP sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor PSU2_+12V_value. Upper Non-recoverable going high (Reading 12.538 .gt Threshold 13.392 Volts)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor PSU2_+12V_value. Lower Non-recoverable going low (Reading 10.548 .lt Threshold 11.232 Volts)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor PSU2_+12V_value. Lower Non-recoverable going low (Reading 12.548 .lt Threshold 11.232 Volts)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "PSU1 Temp_value. Upper Non-recoverable going high (Reading 117 .gt Threshold 115 Degrees)" Motherboard Failure lsta generates a trap for a motherboard failure based on the following conditions for the ipmievd process and with any of { "CPU", "VRD", "PCH", "Inlet", "CHA DIMM 0", "CHA DIMM 1", "CHA DIMM 2", "CHB DIMM 0", "CHB DIMM 1", "CHB DIMM 2", "RAM TMax", "CPU_VCORE", "VBAT", "3VSB", "3VMain", "+5V", "+12V"} in the body of the message: • • • Body of log message contains the text “'Lower Critical going low” and the threshold reported represents an assert condition Body of log message contains the text “Upper Critical going high” and the threshold reported represents an assert condition Body of log message contains the text “Upper Non-recoverable going high” and the threshold reported represents an assert condition. Motherboard failures correspond to the motherboardAttentionNotify NOTIFICATION-TYPE of the SAFENET-APPLIANCE-MIB. Here are examples to generate motherboard traps: sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor VBAT . Lower Critical going low (Reading 1.63 .lt Threshold 2.80 Volts)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Voltage sensor VBAT . Lower Critical going low (Reading 3.30 .lt Threshold 2.80 Volts)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Temperature sensor CPU . Upper Critical going high (Reading 75 .gt Threshold 72 Degrees)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Temperature sensor CPU . Upper Critical going high (Reading 70 .gt Threshold 72 Degrees)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Temperature sensor CPU . Upper Non-recoverable going high (Reading 92 .gt Threshold 89 Degrees)" sysconf snmp trap test -logfacility local4 -loglevel notice -process ipmievd -message "Temperature sensor CPU . Upper Non-recoverable going high (Reading 85 .gt Threshold 89 Degrees)" Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 98 of 102 Luna Monitoring Syslog & SNMP Disk Drive Failure lsta generates a trap for a disk drive failure based on the following conditions for the smartd process: • Severity of the message is “crit”. Disk drive failures correspond to the diskDriveAttentionNotify NOTIFICATION-TYPE of the SAFENET-APPLIANCE-MIB. Use the following text to create a disk drive trap: sysconf snmp trap test -logfacility daemon -loglevel crit -process smartd -message "Device: /dev/sda, Temperature 45 Celsius reached limit of 44 Celsius (Min/Max 31/49)" -pid NTLS Failure lsta generates a trap for an NTLS failure based on the following conditions for the NTLS process: • • Severity of the message is “err”. Severity of the message is “crit”. NTLS failures correspond to the ntlsAttentionNotify NOTIFICATION-TYPE of the SAFENETAPPLIANCE-MIB. Here are examples to create NTLS traps: sysconf snmp trap test -logfacility local5 -loglevel crit -process NTLS -message "error : 0xc0000002 : Unable to create a new connection. " -pid sysconf snmp trap test -logfacility local5 -loglevel crit -process NTLS -message "info : 0 : NTLS CRASH AND BURN! Stack dump saved to /var/log/ntls_bt_2012-02-29_12:05:01" -pid Crypto Failure lsta generates a trap for a crypto failure – the internal HSM for Luna appliances – based on the following conditions: • • For the kernel process, body of log message contains the text “HSM crashed:” For the sysstatd process, body of log message contains the text “30” Crypto failures correspond to the cryptoAttentionNotify NOTIFICATION-TYPE of the SAFENET-APPLIANCE-MIB. Use the following examples to simulate a crypto failure: Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 99 of 102 Luna Monitoring Syslog & SNMP sysconf snmp trap test -logfacility kern -loglevel info -process kernel -message "NOTE: viper0: hsm log: LOG(CRITICAL) HSM crashed:" sysconf snmp trap test -logfacility user -loglevel info -process sysstatd -message "Luna System State Server - OOS Errors: 30,100,60!" Clock Failure lsta generates a trap for an appliance clock failure based on the following conditions: • For the cron process, the year timestamp in any message is prior to 2014. Clock failures correspond to the clockAttentionNotify NOTIFICATION-TYPE of the SAFENETAPPLIANCE-MIB. To create a clock failure trap, use the following commands to set the date back in time and create a condition that results in lsta processing a message: sysconf time 12:05 20120229 sysconf snmp trap test -logfacility authpriv -loglevel debug -process crond -message "pam_unix(crond:account): account root has password changed in future" Remember to set the date and time back to a current value after creating the test message. NOTE: An adjusted clock setting also results in a smartd critical log message that results in a trap: 2012 Feb 29 12:09:29 myLUT daemon crit Resetting next wakeup time. smartd[2478]: System clock time adjusted to the past. Also note that leaving the clock in this state eventually results in real trap events (i.e., no need to use the Lush command to initiate the trap): 2012 Feb 29 12:18:01 myLUT password changed in future authpriv debug crond[25062]: pam_unix(crond:account): account root has Caveats With this release of the Luna appliance, be aware of the following caveats. 1. SafeNet has implemented only a sub-set of possible traps. More will come with subsequent product releases. 2. Enabling traps is an “all-or-nothing” choice. The product does not allow you to configure trap generation for some events at the exclusion of others. 3. NTLS certificate monitoring for expiry is unchanged: the certmonitord process continues to monitor for this event. 4. Only IPMI events (i.e., sensors) support both assert and de-assert events with this release of lsta. Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 100 of 102 Luna Monitoring Syslog & SNMP Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 101 of 102 Luna Monitoring Syslog & SNMP i Example: ATA error count 9 inconsistent with error log pointer 5 Luna Syslog and SNMP Monitoring Guide 007-012417-001 Rev V22 Copyright 2016, Gemalto NV. 102 of 102
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.5 Linearized : Yes Author : Gemalto NV Company : SafeNet Inc. Create Date : 2016:07:18 17:04:15-04:00 Modify Date : 2016:07:18 17:05:10-04:00 Source Modified : D:20160718210110 Language : EN-US Tagged PDF : Yes XMP Toolkit : Adobe XMP Core 5.6-c015 84.158975, 2016/02/13-02:40:29 Metadata Date : 2016:07:18 17:05:10-04:00 Creator Tool : Acrobat PDFMaker 15 for Word Document ID : uuid:1625b40d-15e6-4202-b16c-4c18dba04edd Instance ID : uuid:0dc91762-1c6a-4160-9f4f-aa7d4793c810 Subject : 2 Format : application/pdf Title : Syslog and SNMP Monitoring Guide Creator : Gemalto NV Producer : Adobe PDF Library 15.0 Page Layout : OneColumn Page Count : 102EXIF Metadata provided by EXIF.tools