Hands On Instructions
User Manual:
Open the PDF directly: View PDF .
Page Count: 101
Download | |
Open PDF In Browser | View PDF |
OPENSHIFT HANDS-ON @Microsoft Développer et déployer une application Cloud-Native Guillaume Estrem & Laurent Broudoux AppDev Solution Architect 21 Février 2019 13h00 LAB GUIDE OPENSHIFT HANDS-ON ON AZURE Getting Started Lab 2 Deploying containers from an image Lab 3 Deploying containers from sources Lab 4 Monitoring application health 14h30 16h00 2 Lab 1 PAUSE Lab 5 Distributed Tracing Configuration Lab 6 Getting Application Metrics Lab 7 Azure Service Broker Lab 8 Continuous Delivery 14h45 OPENSHIFT CONCEPTS OVERVIEW A container is the smallest compute unit CONTAINER 4 OPENSHIFT HANDS-ON ON AZURE Containers are created from container images 5 OPENSHIFT HANDS-ON ON AZURE CONTAINER IMAGE CONTAINER BINARY RUNTIME Container images are stored in an image registry IMAGE REGISTRY CONTAINER IMAGE CONTAINER IMAGE CONTAINER IMAGE CONTAINER CONTAINER IMAGE 6 OPENSHIFT HANDS-ON ON AZURE CONTAINER IMAGE CONTAINER IMAGE An image repository contains all versions of an image in the image registry IMAGE REGISTRY myregistry/frontend frontend:latest frontend:2.0 frontend:1.1 frontend:1.0 7 OPENSHIFT HANDS-ON ON AZURE myregistry/mongo CONTAINER IMAGE mongo:latest mongo:3.7 mongo:3.6 mongo:3.4 CONTAINER IMAGE Containers are wrapped in pods which are units of deployment and management POD CONTAINER IP: 10.1.0.11 8 OPENSHIFT HANDS-ON ON AZURE POD CONTAINER CONTAINER IP: 10.1.0.55 Pods configuration is defined in a deployment image name replicas labels cpu memory storage DEPLOYMENT 9 OPENSHIFT HANDS-ON ON AZURE POD POD POD CONTAINER CONTAINER CONTAINER Services provide internal load-balancing and service discovery across pods BACKEND SERVICE role: backend 10 POD POD POD POD CONTAINER CONTAINER CONTAINER CONTAINER role: frontend role: backend role: backend role: backend OPENSHIFT HANDS-ON ON AZURE Apps can talk to each other via services Invoke Backend API BACKEND SERVICE role: backend 11 POD POD POD POD CONTAINER CONTAINER CONTAINER CONTAINER role: frontend role: backend role: backend role: backend OPENSHIFT HANDS-ON ON AZURE Routes add services to the external load-balancer and provide readable urls for the app ROUTE app-prod.mycompany.com > curl http://app-prod.mycompany.com BACKEND SERVICE 12 OPENSHIFT HANDS-ON ON AZURE POD POD POD CONTAINER CONTAINER CONTAINER Projects isolate apps across environments, teams, groups and departments PAYMENT DEV CATALOG POD POD POD C C C ❌ POD POD POD C C C PAYMENT PROD 13 INVENTORY POD POD POD C C C OPENSHIFT HANDS-ON ON AZURE ❌ ❌ POD POD POD C C C LAB 1 Getting started Pick your user ID Go to http://bit.ly/ocp-on-azure and assign your name to a user available. This user will be your identity during the workshop. Don’t use your neighbour user ;) 15 Connect via SSH to the bastion The bastion contains all tools needed for the following workshop. Open your terminal and execute the following command : $ ssh userX@52.143.152.215 For Windows users, download and install Putty : https://www.ssh.com/ssh/putty/windows/install Before starting... ● Make sure you have a userId (userX). Each attendee has its own environment on OpenShift Container Platform ● Fork the GitHub repo https://github.com/lbroudoux/ocp-on-azure-workshop into your own GitHub and clone it in your home directory /home/userX/ on the bastion ● Open a terminal and login into Openshift with the following credentials $ oc login https://masterdnscbmvtdzhvuqye.francecentral.cloudapp.azure.com/ -u userX -p mypassword Login successful. You have access to the following projects and can switch between them with 'oc project': 17 OPENSHIFT HANDS-ON ON AZURE APPLICATION ARCHITECTURE OVERVIEW Grocery Store on OpenShift INVENTO RY 19 OPENSHIFT HANDS-ON ON AZURE CATALOG LAB 2 Deploy containers from an image Create your development environment Let’s go the Web Console ● Via the web console : https://masterdnscbmvtdzhvuqye.francecentral.cloudapp.azure.com ○ Login with the same credentials ○ Create a Project with the following informations ■ Name : fruits-grocery-dev-userX ■ Display Name: UserX - Fruits Grocery - Dev OPENSHIFT HANDS-ON ON AZURE Deploy MongoDB database via the catalog ● Browse the service catalog and search for MongoDB ● Set MongoDb Database Name ○ Name : fruitsdb ● Save and label the deployment config with the command below $ oc label dc/mongodb app=fruits-catalog OPENSHIFT HANDS-ON ON AZURE Check MongoDB deployment One Pod is running. Explore the objects created by OpenShift : image used, TCP port opened and service created OPENSHIFT HANDS-ON ON AZURE Deploy Redis via the CLI Let’s do the deployment of Redis through the CLI rather than the Web console $ oc new-app redis-persistent --name=redis -p DATABASE_SERVICE_NAME=redis -l app=fruits-inventory -n fruits-grocery-dev-userX Quick overview of the command line ● ● ● “redis-persistent” is the template we use from the catalog We specify also a label (app=fruits-inventory ) to select easily all resources related to fruits-inventory in our environment DATABASE_SERVICE_NAME is the service to reach all pods related to Redis OPENSHIFT HANDS-ON ON AZURE Check Redis deployment One Pod is running. Explore the objects created by OpenShift : image used, TCP port opened and service created OPENSHIFT HANDS-ON ON AZURE Grocery Store on OpenShift INVENTO RY 26 OPENSHIFT HANDS-ON ON AZURE CATALOG LAB 3 Deploy containers from source Deploy the fruits catalog with s2i strategy BUILD APP Git Repository code (OpenShift) BUILD IMAGE Developer Source-to-Image (S2I) (OpenShift) DEPLOY (OpenShift) OPENSHIFT HANDS-ON ON AZURE Builder Image Application Container deploy Image Registry Deploy the fruits catalog Our application is developed with Spring Boot. A powerful Java framework to build next-gen application and leverage Openshift capabilities. Let’s use the official Red Hat OpenJDK 8 image Builder to create our container Image from the source code. OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Let’s explore Advanced options to specify environment variables and extras things! OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Complete source code informations to build the SpringBoot app in Openshift ● Name : fruits-catalog ● Context Dir : /fruits-catalog ● Your Git repo URL OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Set environment variables for database credentials and URI MONGODB_USER - pick the right secret MONGODB_PASSWORD - pick the right secret SPRING_DATA_MONGODB_URI : mongodb://${MONGODB_USER}:${MONGODB_PASSWORD}@mongodb:27017/frui tsdb MongoDB credentials are located in a secret named mongodb ● ● ● OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Explore the application resources deployed Container Image, Build used, ports, routes ... Click on the blue circle to explore the pod instance OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Explore the pod configuration ● Check Environment variables ● Access to the terminal ● Explore application logs ● Visualize metrics OPENSHIFT HANDS-ON ON AZURE Deploy the fruits catalog Test the fruits catalog Insert fruits in your catalog microservices via the fruits-catalog API $ curl `oc get route/fruits-catalog -o template --template={{.spec.host}}`/api/fruits -XPOST -H "Content-Type: application/json" -d '{"name":"Orange", "origin":"Spain"}' $ curl `oc get route/fruits-catalog -o template --template={{.spec.host}}`/api/fruits -XPOST -H "Content-Type: application/json" -d '{"name":"Apple", "origin":"France"}' Get all fruits from the fruits-catalog $ curl `oc get route/fruits-catalog -o template --template={{.spec.host}}`/api/fruits -v OPENSHIFT HANDS-ON ON AZURE Deploy the Fruits inventory We use the same s2i strategy to build and deploy the app from source code Choose the Node.JS image builder from the catalog OPENSHIFT HANDS-ON ON AZURE Deploy the Fruits inventory ● Name : fruits-inventory ● Context Dir : /fruits-inventory ● Your Git Repo URL OPENSHIFT HANDS-ON ON AZURE Deploy the Fruits inventory Set environment variables to access Redis Cache component already containerized ● ● ● REDIS_HOST - redis REDIS_PASSWORD - pick the right secret FRUITS_CATALOG_HOST - fruits-catalog Set environment variables as described OPENSHIFT HANDS-ON ON AZURE Deploy the Fruits inventory Check that the component works properly with Redis cache Let’s get all fruits in the Grocery Store with their quantity $ curl `oc get route/fruits-inventory -o template --template={{.spec.host}}`/api/fruits [{"id":"5c641f4d18909600016320d0","name":"Orange","origin":"Spain","quantity":"1230"},{"id":" 5c64225818909600016320d1","name":"Apple","origin":"France","quantity":"356"}] You can also explore the deployment and the pod resources OPENSHIFT HANDS-ON ON AZURE LAB 4 Monitoring application health LAB 4: Monitoring Application Health 41 ● Review Health endpoints in services ● Add health probes to inventory-service ● Add health probes to shop-ui front-end ● Explore pod metrics OPENSHIFT HANDS-ON ON AZURE HEALTH PROBES PROBE TYPES Is it ready? Is it alive? CONTAINER 42 OPENSHIFT HANDS-ON ON AZURE CONTAINER CONTAINER PROBE CHECKS HTTP Shell Command TCP Port Health probes There are two type of health probes available in OpenShift: liveness probes and readiness probes. Liveness probes are to know when to restart a container and readiness probes to know when a Container is ready to start accepting traffic. Health probes also provide crucial benefits when automating deployments with practices like rolling updates in order to remove downtime during deployments. A readiness health probe would signal OpenShift when to switch traffic from the old version of the container to the new version so that the users don’t get affected during deployments. 43 OPENSHIFT HANDS-ON ON AZURE Add Health check to fruits catalog We can do it through the web console or the CLI Set HTTP request to check readiness. An endpoint is already defined in the fruits catalog. We use business and technical endpoints provided natively by the actuator Spring Boot library. This library will be used in others labs ;) OPENSHIFT HANDS-ON ON AZURE Add health check to Fruits catalog Add the liveness probe Is the app still running ? We use the same endpoint as the readiness for this example. OPENSHIFT HANDS-ON ON AZURE Save and check rolling upgrade strategy Click Save and then click the Overview button in the left navigation. You will notice that fruits-catalog pod is getting restarted and it stays light blue for a while. This is a sign that the pod(s) have not yet passed their readiness checks and it turns blue when it’s ready! 46 OPENSHIFT HANDS-ON ON AZURE Add health checks with the CLI We set an HTTP Request for both health checks $ oc set probe dc/fruits-inventory --liveness --get-url=http://:8080/api/health/liveness --initial-delay-seconds=60 --period-seconds=30 $ oc set probe dc/fruits-inventory --readiness --get-url=http://:8080/api/health/readiness OPENSHIFT HANDS-ON ON AZURE Monitoring pod metrics Metrics are another important aspect of monitoring applications which is required in order to gain visibility into how the application behaves and particularly in identifying issues. OpenShift provides container metrics out-of-the-box and displays how much memory, cpu and network each container has been consuming over time. In the project overview, you can see three charts near each pod that shows the resource consumption by that pod. 48 OPENSHIFT HANDS-ON ON AZURE Monitoring pod metrics 49 OPENSHIFT HANDS-ON ON AZURE LAB 5 Distributed tracing configuration LAB 5: Distributed tracing configuration 51 ● Externalize and manage application configuration ● Add Jaeger configuration to fruits-catalog ● Explore distributed traces OPENSHIFT HANDS-ON ON AZURE What is distributed tracing ? Spans OpenTracing instrumentation Relationships 52 OPENSHIFT HANDS-ON ON AZURE Distributed tracing Jaeger is an OpenTracing implementation and is available in the Cockpit environment. 53 OPENSHIFT HANDS-ON ON AZURE Add Jaeger configuration to fruits-catalog Before setting Jaeger in the fruits-catalog application, we have to add a specific role to the current project to view particular objects, especially ConfigMap ... $ oc policy add-role-to-user view -n $(oc project -q) -z default OPENSHIFT HANDS-ON ON AZURE ConfigMap in OpenShift ● Config maps inject config data into containers ● Config maps can hold ○ ○ ● Containers see config maps as ○ ○ ● Files on the filesystem Environment variables Secrets are like config maps for sensitive data ○ 55 Properties (key-value pairs) Files (JSON, XML, etc) Credentials, certificates, SSH keys, etc OPENSHIFT HANDS-ON ON AZURE Configuration management Pod Pod Pod RUNTIME Injected in container memory CONFIGURATION FOO=foo BAR=bar VARIABLES D’ENVIRONNEMENT 56 OPENSHIFT HANDS-ON ON AZURE Volume Mounted on filesystem in read-only application.properties com.svc=http://svc … … CONFIGMAPS Volume De-cyphered and mounted on filesystem in read-only mykey.p12 databasecredentials.properties SECRETS Add Jaeger configuration to fruits-catalog Create a configMap with the CLI $ cd fruits-catalog $ oc create configmap fruits-catalog-config --from-file=application.yml Edit ConfigMap ( Actions > Edit Yaml ) created and set Jaeger host as : jaeger-agent.cockpit.svc.cluster.local Click Add to Application Now pod is redeploying 57 OPENSHIFT HANDS-ON ON AZURE Add Jaeger host to fruits-inventory A Jaeger tracer is already set for all invocations in fruits-inventory. We set the Jaeger host as environment variable $ oc set env dc/fruits-inventory JAEGER_HOST=jaeger-agent.cockpit.svc.cluster.local A new deployment is created. Get all fruits with their stock through the fruits-inventory API and jump to Jaeger to see the detailed trace $ curl `oc get route/fruits-catalog -o template --template={{.spec.host}}`/api/fruits -v 58 OPENSHIFT HANDS-ON ON AZURE Explore Distributed Traces with Jaeger Go to Jaeger console via https://bit.ly/2BKWuTN Jaeger is deployed on Openshift in an other project name cockpit. 59 OPENSHIFT HANDS-ON ON AZURE Filter the right Jaeger trace As we use a mutual Jaeger, you need to filter on your pod fruits-catalog hostname $ oc get pods -l app=fruits-catalog NAME READY STATUS fruits-catalog-4-4phqn 1/1 Running RESTARTS 0 AGE 40m Following filter criterias are : ● ● ● Services : fruits-inventory Operation: / Tags: hostname=fruits-catalog-4-4phqn Click on Find Traces OPENSHIFT HANDS-ON ON AZURE Explore the Jaeger trace OPENSHIFT HANDS-ON ON AZURE LAB 6 Getting application metrics LAB 6: GETTING APPLICATION METRICS 63 ● Update Prometheus configuration ● Add Prometheus datasource in Grafana OPENSHIFT HANDS-ON ON AZURE Prometheus monitoring OpenShift now provides Prometheus templates for automated deployment. One instance is available into a cockpit project. A Grafana instance on same project. 64 OPENSHIFT HANDS-ON ON AZURE Prometheus monitoring For a quick run, we’ll use JMX Exporter Prometheus Agent that expose JMX metrics as Prometheus endpoints. This is already configured into fruits-catalog thanks to actuator library. A middleware Prometheus is added in fruits-inventory 65 OPENSHIFT HANDS-ON ON AZURE Check Prometheus metrics in deployed pods Access to the pod terminal with oc rsh command $ oc rsh dc/fruits-catalog # Now logging in fruits-catalog pod $ curl http://localhost:8080/actuator/prometheus … # TYPE jvm_buffer_total_capacity_bytes gauge jvm_buffer_total_capacity_bytes{id="direct",} 82807.0 jvm_buffer_total_capacity_bytes{id="mapped",} 0.0 … $ curl http://localhost:8080/actuator/metrics # display metrics available {"names":["jvm.memory.max","jvm.threads.states","process.files.max", "jvm.gc.memory.promoted" ... OPENSHIFT HANDS-ON ON AZURE Check Prometheus console now ... Go to Prometheus console : https://prometheus-cockpit.52.143.158.219.nip.io/ in the target menu Nothing is sent by fruits-catalog and fruits-inventory Prometheus console ! Prometheus scraps by default /metrics endpoint on port 9900. Our 2 back-ends expose a different Prometheus endpoint. We need to annotate our application Kubernetes services to be discovered by Prometheus $ oc annotate service/fruits-catalog prometheus.io/scrape=true prometheus.io/path=/actuator/prometheus prometheus.io/port=8080 $ oc annotate service/fruits-inventory prometheus.io/scrape=true prometheus.io/port=8080 OPENSHIFT HANDS-ON ON AZURE Import Grafana Dashboard Grafana URL : https://grafana-cockpit.52.143.158.219.nip.io/ Click on New Dashboard and Import Dashboard Copy and paste the following json: https://raw.githubusercontent.com/lbroudoux/ocp-on-azure-workshop/master/grafana-d ashboard-user0.json OPENSHIFT HANDS-ON ON AZURE Import Grafana Dashboard Change Dashboard name with your user ID OPENSHIFT HANDS-ON ON AZURE Grafana Dashboard example 70 OPENSHIFT HANDS-ON ON AZURE LAB 7 Azure Service Broker Why a service broker ? ☑ Open ticket ☑ Wait for allocation ☑ Receive credentials ☑ Add to app ☑ Deploy app SERVICE CONSUMER SERVICE PROVIDER Manual, Time-consuming and Inconsistent 72 OPENSHIFT TECHNICAL OVERVIEW A multi-vendor project to standardize how services are consumed on cloud-native platforms across service providers 73 OPENSHIFT TECHNICAL OVERVIEW What is a service broker ? SERVICE CONSUMER SERVICE CATALOG SERVICE BROKER Automated, Standard and Consistent 74 OPENSHIFT TECHNICAL OVERVIEW SERVICE PROVIDER OpenShift service catalog ANSIBLE AWS Service Broker AWS Other Service Brokers OTHER COMPATIBLE SERVICES er OPENSHIFT TECHNICAL OVERVIEW OpenShift Ansible Broker re ok zu r A eB ic rv 75 OPENSHIFT Se OPENSHIFT SERVICE CATALOG OpenShift Template Broker OpenShift Templates Ansible Playbook Bundles AWS Services Other Services OPEN SERVICE BROKER AZURE https://github.com/Azure/open-service-broker-azure Supported services ● ● ● ● ● ● ● ● ● ● ● 76 Azure Container Instances Azure CosmosDB Azure Database for MySQL Azure Database for PostgreSQL Azure Event Hubs Azure Key Vault Azure Redis Cache Azure SQL Database Azure Search Azure Service Bus Azure Storage Grocery Store on OpenShift and Azure OpenShift INVENTO RY CATALO G Open Service Broker Azure Azure 77 OPENSHIFT HANDS-ON ON AZURE Create your production environment Let’s go the Web Console ● Via the web console : https://masterdnscbmvtdzhvuqye.francecentral.cloudapp.azure.com ○ Login with the same credentials ○ Create a Project with the following informations ■ Name : fruits-grocery-prod-userX ■ Display Name: UserX - Fruits Grocery - Prod OPENSHIFT HANDS-ON ON AZURE Deploy a Redis Cache instance with Open Service Broker Azure 79 Deploy Redis Cache DB with OSBA Complete the following settings ● Select a Plan ○ Basic Tier ● Configuration ○ location : eastus ○ resourceGroup : osba ● Bindings ○ Don’t bind to secrets. We will do it Manually :) OPENSHIFT HANDS-ON ON AZURE Let’s go to the backstage A Redis Cache instance has been provisioned in Azure through the Azure Service Broker OPENSHIFT HANDS-ON ON AZURE Deploy a Cosmo DB instance with OSBA 82 Deploy a Cosmo DB with OSBA Complete the following settings Configuration ● ● ● ● defaultConsistencyLevel = Session allowedIPRanges = 0.0.0.0/0 . Then click Add and then click the X Location : eastus resourceGroup : osba Binds: ● ● Add secrets bindings Service Broker will retrieve credentials CosmoDB instance from Azure Our two services provisioned ! Redis and CosmoDB are provisioned asynchronously in Azure via the Open Service Broker. You can consume both services through OpenShift via the binding mechanism. *Due to OSBA implementation Redis stays in Pending status. LAB 8 Continuous delivery LAB 8: Automating Deployments Using Tags and Pipelines 86 ● Prepare a Production environment ● Explore the deployment configurations ● Promote images to production ● Create an OpenShift Jenkins Pipeline ● Add a Webhook to run the pipeline on every code change ● Change some code and review OPENSHIFT HANDS-ON ON AZURE Deployment pipeline physical virtual private cloud dev source repository CI/CD engine container public cloud 87 OPENSHIFT HANDS-ON ON AZURE Rolling Upgrades Blue/Green Deployments A/B Testing CI/CD with OpenShift DEV INT QA PROD Image build process Orchestrateur 88 OPENSHIFT HANDS-ON ON AZURE OpenShift Pipelines 89 ● CI/CD workflow via Jenkins ● Pipelines are started, monitored, and managed similar to other builds ● Auto-provisioning of Jenkins server ● On-demand Jenkins slaves ● Embedded Jenkinsfile or in Git repo OPENSHIFT HANDS-ON ON AZURE pipeline { agent { label 'maven' } stages { stage('build app') { steps { git url: 'https://git/app.git' sh "mvn package" } } stage('build image') { steps { script { openshift.withCluster() { openshift.startBuild("...") } } } } } } Create Redis Cache secrets ● Go back to the spreadsheet : https://bit.ly/2TWsI5D ● Update REDIS_HOST and REDIS_PASSWORD environment variables from prepare_prod.sh file with the values from the spreadsheet $ vi /home/userX/prepare-prod.sh OPENSHIFT HANDS-ON ON AZURE Prepare a Production environment A wrap-up script has been prepared for you. It will contains all resources created previously in the Development project. $ ./prepare-prod.sh 91 OPENSHIFT HANDS-ON ON AZURE Explore the deployment configurations From overview on web console, check the deployment configuration All deployment are cancelled. 92 OPENSHIFT HANDS-ON ON AZURE Explore the deployment configurations Clicking on a deployment configuration, you should see that there’s no automatic trigger defined for deployment. You shall also notice that the image used for deployment is coming from your development project ! 93 OPENSHIFT HANDS-ON ON AZURE Explore the deployment configurations Access detailed configuration by choosing Edit in Actions menu. Check that the image referenced into your dev project has the :promoteToProd tag. Because this tag does not exists, deployment will fail ! 94 OPENSHIFT HANDS-ON ON AZURE Promote images to production The wrap-up script can be used again here through a new command. The command will tag all images from development streams and rollout all the deployments. $ . /home/userX/deploy-prod.sh Tag fruits-grocery-dev-user0/fruits-catalog:promoteToProd set to fruits-grocery-dev-user0/fruits-catalog@sha256:5eeb089a5df9aa55b4e80c581014a674c1e2f7e902c92a3f5c48e0df4155e95 7. Tag fruits-grocery-dev-user0/fruits-inventory:promoteToProd set to fruits-grocery-dev-user0/fruits-inventory@sha256:29a17627c330a5568f6a956ffddc5f7c3e17ab4839e22085899b7eb0328 9705a. deploymentconfig "fruits-catalog" rolled out deploymentconfig "fruits-inventory" rolled out 95 OPENSHIFT HANDS-ON ON AZURE Promote images to production Check deployment are successful ! But wait … we have also created a pipeline. Just go to your development project. 96 OPENSHIFT HANDS-ON ON AZURE Create an OpenShift Jenkins Pipeline In your development project within the Builds section, Pipelines subsection, check that inventory-service-pipeline has been created. Triggers with webhooks provide a full developer experience to automate its deployment from a local env to production environment 97 OPENSHIFT HANDS-ON ON AZURE Start your Jenkins pipeline We deploy the fruits-inventory application from Dev to Prod with complex tests ... Check Jenkins pipeline job logs via the Jenkins console. Click on “View Log” OPENSHIFT HANDS-ON ON AZURE CONGRATULATIONS ! YOU’RE A CLOUD-NATIVE APPS DEVELOPER. 99 CONTAINERS & CLOUD-NATIVE ROADSHOW LEARN.OPENSHIFT.COM 100 OPENSHIFT HANDS-ON @ Société Générale THANK YOU plus.google.com/+RedHat facebook.com/redhatinc linkedin.com/company/red-hat twitter.com/RedHatNews youtube.com/user/RedHatVideos
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.4 Linearized : No Warning : Root object (4 0 obj) not found at 16EXIF Metadata provided by EXIF.tools