CSI Driver For VxFlex OS Product Guide Vx Flex

User Manual:

Open the PDF directly: View PDF .
Page Count: 20

Download
Open PDF In Browser	View PDF

CSI Driver for VxFlex OS
Version 1.0

Product Guide
302-005-685
01

Copyright © 2019 Dell Inc. or its subsidiaries. All rights reserved.
Published April 2019
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND
WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED
IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
Published in the USA.
Dell EMC
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 In North America 1-866-464-7381
www.DellEMC.com

CSI Driver for VxFlex OS 1.0 Product Guide

CONTENTS

Chapter 1

Introduction

Product overview......................................................................................... 6

Chapter 2

Installing the CSI Driver for VxFlex OS

Installation overview..................................................................................... 8
Prerequisites................................................................................................ 8
Enable Kubernetes feature gates.....................................................8
Configure Docker service............................................................... 10
Install Helm package manager........................................................ 10
Install VxFlex OS Storage Data Client............................................. 11
Install CSI Driver for VxFlex OS................................................................... 11

Chapter 3

Testing driver

Test deploying a simple pod with VxFlex OS storage...................................16
Test creating snapshots.............................................................................. 17
Test restoring from a snapshot................................................................... 18

Chapter 4

Troubleshooting

Troubleshooting......................................................................................... 20

CSI Driver for VxFlex OS 1.0 Product Guide

CONTENTS

CSI Driver for VxFlex OS 1.0 Product Guide

CHAPTER 1
Introduction

Product overview.................................................................................................6

Introduction

Product overview
The CSI Driver for VxFlex OS is a plug-in that is installed into Kubernetes to provide
persistent storage using Dell EMC VxFlex OS storage system.
The CSI Driver for VxFlex OS and Kubernetes communicate using the Container
Storage Interface protocol. CSI Driver for VxFlex OS conforms to CSI specification
v1.0 and compatible with Kubernetes versions 1.13.1, 1.13.2 and 1.13.3.
Features of CSI Driver for VxFlex OS
The CSI Driver for VxFlex OS supports the following features:
l

Persistent volume (PV) capabilities - create, list, delete, and create-fromsnapshot

Dynamic and static PV provisioning

Snapshot capabilities - create, delete, and list

Volume mount as ext4 or xfs file system on the worker node

Supports the following access modes:
n

multi-node-single-writer

single-node-writer

single-node-reader-only

multi-node-reader-only

multi-node-single-writer

Volume prefix for easy LUN identification

Supports HELM charts installer

Supports CSI 1.0

Supports Kubernetes version 1.13.1, 1.13.2 and 1.13.3

Compatible with VxFlex OS version 3.0 with medium granularity storage pools.

The CSI Driver for VxFlex OS depends on the following libraries:
l

gofsutil

gocsi

goscaleio

godog

protobuf

logrus

gosync

grpc

These libraries are used in the current version of the driver.

CSI Driver for VxFlex OS 1.0 Product Guide

CHAPTER 2
Installing the CSI Driver for VxFlex OS

This chapter contains the following sections:
l
l
l

Installation overview............................................................................................ 8
Prerequisites........................................................................................................ 8
Install CSI Driver for VxFlex OS........................................................................... 11

Installing the CSI Driver for VxFlex OS

Installation overview
The Helm chart installs CSI Driver for VxFlex OS using a shell script. This script installs
the CSI driver container image along with the required Kubernetes sidecar containers.
The controller section of the Helm chart installs the following components in a
Stateful Set in the namespace vxflexos:
l

CSI Driver for VxFlex OS

Kubernetes Provisioner, which provisions the provisioning volumes

Kubernetes Attacher, which attaches the volumes to the containers

Kubernetes Snapshotter, which provides snapshot support

The node section of the Helm chart installs the following component in a Daemon Set
in the namespace vxflexos:
l

CSI Driver for VxFlex OS

Kubernetes Registrar, which handles the driver registration

Prerequisites
Before you install CSI Driver for VxFlex OS, verify the requirements that are
mentioned in this topic are installed and configured.
Requirements
l

Install Kubernetes. The CSI Driver for VxFlex OS works with Kubernetes version
1.13.1 or later.

Enable the Kubernetes feature gates

Configure Docker service

Install Helm and Tiller with a service account

Install VxFlex OS SDC

Enable Kubernetes feature gates
The Kubernetes feature gates must be enabled before installing CSI Driver for VxFlex
OS .
The Feature Gates section of Kubernetes home page lists the Kubernetes feature
gates. The following Kubernetes feature gates must be enabled:

VolumeSnapshotDataSource

KubeletPluginsWatcher

CSINodeInfo

CSIDriverRegistry

BlockVolume

CSIBlockVolume

CSI Driver for VxFlex OS 1.0 Product Guide

Installing the CSI Driver for VxFlex OS

Procedure
1. On each master and node of Kubernetes, edit /var/lib/kubelet/config.yaml and
add the following lines at the end to set feature-gate settings for the kubelets:
/var/lib/kubelet/config.yaml
VolumeSnapshotDataSource: true
KubeletPluginsWatcher: true
CSINodeInfo: true
CSIDriverRegistry: true
BlockVolume: true
CSIBlockVolume: true

2. On the master, set the feature gate settings of the kube-apiserver.yaml file as
follows:
/etc/kubernetes/manifests/kube-apiserver.yaml - --featuregates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true
,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIB
lockVolume=true

3. On the master, set the feature gate settings of the kube-controllermanager.yaml file as follows:
/etc/kubernetes/manifests/kube-controller-manager.yaml - -featuregates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true
,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIB
lockVolume=true

4. On the master, set the feature gate settings of the kube-scheduler.yaml file as
follows:
/etc/kubernetes/manifests/kube-scheduler.yaml - --featuregates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true
,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIB
lockVolume=true

5. On each node, edit the variable KUBELET_KUBECONFIG_ARGS of /etc/
systemd/system/kubelet.service.d/10-kubeadm.conf file as
follows:
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrapkubeconfig=/etc/kubernetes/bootstrap-kubelet.conf -kubeconfig=/etc/kubernetes/kubelet.conf --allowprivileged=true --featuregates=VolumeSnapshotDataSource=true,KubeletPluginsWatcher=true
,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIB
lockVolume=true"

6. Restart the kublet with systemctl daemon-reload and systemctl
restart kubelet on all nodes.

Enable Kubernetes feature gates

Installing the CSI Driver for VxFlex OS

Configure Docker service
The mount propagation in Docker must be configured on all Kubernetes nodes before
installing CSI Driver for VxFlex OS.
Procedure
1. Edit the service section of /etc/systemd/system/multiuser.target.wants/docker.service file as follows:
docker.service
[Service]
...
MountFlags=shared

2. Restart the docker service with systemctl daemon-reload and
systemctl restart docker on all the nodes.

Install Helm package manager
Install the Helm and Tiller package manager on the master node.
To install Helm, run the following commands:
Procedure
1. Run curl https://raw.githubusercontent.com/helm/helm/
master/scripts/get > get_helm.sh
2. Run chmod 700 get_helm.sh
3. Run ./get_helm.sh
4. Run helm init
5. Run helm version to test the Helm installation.
6. Set up a service account for Tiller as follows:
a. Create a rbac-config.yaml file and add the following snippet to the file:
apiVersion: v1
kind: ServiceAccount
metadata:
name: tiller
namespace: kube-system
--kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: tiller-clusterrolebinding
subjects:
- kind: ServiceAccount
name: tiller
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: ""

b. Run kubectl create -f rbac-config.yaml to create the service
account.
10

CSI Driver for VxFlex OS 1.0 Product Guide

Installing the CSI Driver for VxFlex OS

7. Apply the service account to Tiller using helm init --upgrade -service-account tiller

Install VxFlex OS Storage Data Client
Install the VxFlex OS Storage Data Client (SDC) on all Kubernetes nodes.
For detailed VxFlex OS installation procedure, see the Dell EMC VxFlex OS Deployment
Guide. Install the VxFlex OS SDC as follows:
Procedure
1. Download the VxFlex OS SDS from Dell EMC Online support. The filename is
EMC-ScaleIO-sdc-*.rpm, where * is the SDC name corresponding to the
VxFlex OS installation version.
2. Export the shell variable MDM_IP in a comma-separated list. This list contains
the IP addresses of the MDMs.
export MDM_IP=10.254.16.58,10.254.16.62
3. Install the SDC with the command rpm -iv ./EMC-ScaleIO-sdc*.x86_64.rpm, where * is the SDC name corresponding to the VxFlex OS
installation version.

Install CSI Driver for VxFlex OS
Install CSI Driver for VxFlex OS using this procedure.
Before you begin
l

You must have the downloaded files, including the Helm chart from the source
repository at github.com/dell/csi-vxflexos, ready for this procedure.

In the top level helm directory, there should be two shell scripts,
install.vxflexos and uninstall.vxflexos. These scripts handle some of
the pre and post operations that cannot be performed in the helm chart, such as
creating Custom Resource Definitions (CRDs), if needed.

You must create a Kubernetes secret with your VxFlex OS username and
password. Use the secret.yaml file to create the secret with the following values
to match the default installation parameters:
n

Name: vxflexos-creds

Namespace: vxflexos

Note
l

Create the namespace using kubectl create namespace vxflexos.

For more information about creating a Kubernetes secret, see: Kubernetes
documentation: Overview of Secrets

Procedure
1. Collect information from theVxFlex OS SDC (Storage Data Client) by executing
the get_vxflexos_info.sh script located in the top-level helm directory.
This script shows the VxFlex OS system ID and MDM IP addresses. Make a note
of the value for these parameters as they must be entered in the myvalues.yaml
file.
Install VxFlex OS Storage Data Client

Installing the CSI Driver for VxFlex OS

Note

Your SDC might have multiple VxFlex OS systems registered. Ensure that you
choose the correct values.
2. Copy the csi-vxflexos/values.yaml into a file in the same directory as
the install.vxflexos named myvalues.yaml, to customize settings for installation.
3. Edit myvalues.yaml to set the following parameters for your installation:
l

Set the systemName string variable to the VxFlex OS system name or
system ID. This value was obtained by running the
get_vxflexos_info.sh script in Step 1 of this procedure.
Set the restGateway string variable to the URL of your system’s REST API
Gateway. You can obtain this value from the VxFlex OS administrator.
Set the storagePool string variable to a default (already existing) storage
pool name in your VxFlex OS system.
Note

New storage pools can be created in VxFlex OS UI and CLI utilities.
l

Set the mdmIP string variable to a comma separated list of MDM IP
addresses.
Set the volumeNamePrefix string variable so that volumes created by the
driver have a default prefix. If one VxFlex OS system is servicing several
different Kubernetes installations or users, these prefixes help you
distinguish them.
The controllerCount variable is used by advanced users to deploy multiple
controller instances. The specified default value 1 is designed to work as
expected. You can modify the value of this variable to set the desired
number of CSI controller replicas.
Set the enablelistvolumesnapshot variable false unless instructed
otherwise, by Dell EMC support. It causes snapshots to be included in the
CSI operation ListVolumes.
The Helm charts create a Kubernetes StorageClass while deploying CSI
Driver for VxFlex OS. The StorageClass section includes following variables:
n

The name string variable defines the name of the Kubernetes storage
class that the Helm charts will create. For example, the "vxflexos” base
name will be used to generate names such as " vxflexos" and "vxflexosxfs".
The reclaimPolicy string variable defines whether the volumes will be
retained or deleted when the assigned pod is destroyed. The valid values
for this variable are Retain or Delete.
The isDefault variable (valid values for this variable are true or false)
will set the newly created storage class as default forKubernetes.

CSI Driver for VxFlex OS 1.0 Product Guide

Installing the CSI Driver for VxFlex OS

Note

– Set this value to “true” only if you expect VxFlex OS to be your
principle storage provider, as it will be used in PersitentVolumeClaims
where no storageclass is provided. After installation, you can add
custom storage classes if desired.
– All strings must be contained within double quotes.
4. Run the sh install.vxflexos command to proceed with the installation.
A successful installation should emit messages that look similar to the following
samples:
[root@k8s113a-10-247-102-211 helm]# sh install.vxflexos
NAME:
vxflexos
LAST DEPLOYED: Thu Jan 24 18:28:35 2019
NAMESPACE: vxflexos
STATUS: DEPLOYED
… omitted lots here …
NAME
READY
STATUS
vxflexos-controller-0
4/4
Running
vxflexos-node-r5kdt
2/2
Running
vxflexos-node-tq5tj
2/2
Running
CSIDrivers:
NAME
AGE
vxflexos
21s
CSINodeInfos:
NAME
AGE
k8s113a-10-247-102-213.lss.emc.com
21h
k8s113a-10-247-102-215.lss.emc.com
21h
StorageClasses:
NAME
PROVISIONER
AGE
vxflexos (default)
csi-vxflexos
21s
vxflexos-xfs
csi-vxflexos
21s
VolumeSnapshotClasses:
NAME
AGE
vxflexos-snapclass
2d

RESTARTS
0
0
0

AGE
20s
20s
20s

Results
At the end of the script, the kubectl get pods -n vxflexos is called to GET
the status of the pods and you will see the following:
l

Vxflex-controller-0 with 4/4 containers ready, and status displayed as Running.

Agent pods with 2/2 containers and the status displayed as Running.

Finally, the script lists the created storageclasses such as, "vxflexos" and "vxflexosxfs". Additional storage classes can be created for different combinations of file
system types and VxFlex OS storage pools. The script also creates
volumesnapshotclasses such as, "vxflexos-snapclass" and other snapshots classes.

Install CSI Driver for VxFlex OS

Installing the CSI Driver for VxFlex OS

CSI Driver for VxFlex OS 1.0 Product Guide

CHAPTER 3
Testing driver

This chapter contains the following sections:
l
l
l

Test deploying a simple pod with VxFlex OS storage.......................................... 16
Test creating snapshots......................................................................................17
Test restoring from a snapshot...........................................................................18

Testing driver

Test deploying a simple pod with VxFlex OS storage
Test the deployment workflow of a simple pod on VxFlex OS storage.
Before you begin
In the source code, there is a directory that contains examples of how you can use the
driver. To use these examples, you must create a "test" namespace, using "kubectl
create namespace test", before you can start testing.
The starttest.sh script is located in the csi-vxflexos/test/helm directory. This
script is used in the following procedure to deploy helm charts that test the
deployment of a simple pod.
Procedure
1. Navigate to the test/helm directory, which contains the starttest.sh and the
2vols directories.
This directory contains a simple Helm chart that will deploy a pod that uses two
VxFlex OS volumes.
2. Run the sh starttest.sh 2vols command to deploy the pod.
You should see the following:
Normal
Pulled
38s
kubelet,
k8s113a-10-247-102-215.lss.emc.com Successfully pulled image
"docker.io/centos:latest"
Normal
Created
38s
kubelet, k8s113a-10-247-102-215.lss.emc.com Created container
Normal
Started
38s
kubelet, k8s113a-10-247-102-215.lss.emc.com Started container
/dev/scinib
8125880
36852
7653216
1% /data0
/dev/scinia
16766976
32944 16734032
1% /data1
/dev/scinib on /data0 type ext4 (rw,relatime,data=ordered)
/dev/scinia on /data1 type xfs
(rw,relatime,attr2,inode64,noquota)

3. To stop the test, run sh stoptest.sh.
This script will delete the pods and the volumes depending on the retention
setting you have configured.
Results
An outline of this workflow is described below:
1. The 2vols helm chart contains two PersistentVolumeClaim definitions, one in
pvc0.yaml, and the other in pvc1.yaml. They are referenced by the test.yaml which
creates the pod. The contents of the Pvc0.yaml file are described below:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvol0
namespace: test
spec:
accessModes:

CSI Driver for VxFlex OS 1.0 Product Guide

Testing driver

- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 8Gi
storageClassName: vxflexos

2. The volumeMode: Filesystem requires a mounted file system and the
resources.requests.storage of 8Gi requires an 8 GB file. In this case, the
storageClassName: vxflexos directs the system to use one of the pre-defined
storage classes created by the CSI Driver for VxFlex OS installation process. This
step yields a mounted ext4 file system. You can see the storage class definitions in
the VxFlex OS installation helm chart files storageclass.yaml and storageclassxfs.yaml.
3. If you compare pvol0.yaml and pvol1.yaml, you will find that the latter uses a
different storage class; vxflexos-xfs. This class gives you an xfs file system.
4. To see the volumes you created, run kubectl get persistentvolumeclaim
–n test and kubectl describe persistentvolumeclaim –n test.
Note

For more information about Kubernetes objects like StatefulSet and
PersistentVolumeClaim see Kubernetes documentation: Concepts.

Test creating snapshots
Test the workflow for snapshot creation.
Procedure
1. Start the 2vols container and leave it running.
2. Run the snaptest.sh shell script.
This will create a snapshot of each of the volumes in the container using
VolumeSnapshot objects defined in snap1.yaml and snap2.yaml. The following
are the contents of snap1.yaml:
apiVersion: snapshot.storage.k8s.io/v1alpha1
kind: VolumeSnapshot
metadata:
name: pvol0-snap1
namespace: test
spec:
snapshotClassName: vxflexos-snapclass
source:
name: pvol0
kind: PersistentVolumeClaim

Results
The snaptest.sh script will create a snapshot using the definitions in the snap1.yaml
file. The spec.source section contains the volume that will be snapped. For example, if
the volume to be snapped is pvol0, then the created snapshot is named pvol0-snap1.
Notice that this VolumeSnapshot class has a reference to a snapshotClassName:
vxflexos-snapclass. The CSI Driver for VxFlex OS installation creates this class as its
default snapshot class. You can see its definition in the installation directory file
volumesnapshotclass.yaml.
Test creating snapshots

Testing driver

Note

The snaptest.sh shell script creates the snapshots, describes them, and then deletes
them. You can see your snapshots using kubectl get volumesnapshot -n test.

Test restoring from a snapshot
Test the restore operation workflow to restore from a snapshot.
Before you begin
Ensure that you have stopped any previous test instance before performing this
procedure.
To test the restore operation from a snapshot:
Procedure
1. Run the snaprestoretest.sh shell script.
This script deploys the 2vols example, creates a snap of pvol0, and then
updates the deployed helm chart from the updated directory 2vols+restore.
This then adds an additional volume that is created from the snapshot.
Results
An outline of this workflow is described below:
1. The snapshot is taken using snap1.yaml.
2. Helm is called to upgrade the deployment with a new definition, which is found in
the 2vols+restore directory. The helm/templates directory contains the newly
created createFromSnap.yaml file. The script then creates a
PersistentVolumeClaim, which is a volume that is dynamically created from the
snapshot. Then the helm deployment is upgraded to contain the newly created
third volume. In other words, when the snaprestoretest.sh creates a new volume
with data from the snapshot, the restore operation is tested. The contents of the
createFromSnap.yaml are described below:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: restorepvc
namespace: test
spec:
storageClassName: vxflexos
dataSource:
name: pvol0-snap1
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi

Note

The spec.dataSource clause, specifies a source "VolumeSnapshot" named pvol0snap1 which matches the snapshot's name in snap1.yaml.

CSI Driver for VxFlex OS 1.0 Product Guide

CHAPTER 4
Troubleshooting

This chapter contains the following topics:
l

Troubleshooting................................................................................................. 20

Troubleshooting

Troubleshooting
The following table lists the CSI Driver for VxFlex OS installation troubleshooting
scenarios:
Symptoms

Prevention, resolution, or workaround

The installation fails with the following error
message:

Install the VxFlex OS SDC on listed nodes.
The SDC must be installed on all the
Kubernetes master and nodes.

Node xxx does not have the SDC
installed

When you run the command kubectl
describe pods vxflexoscontroller-0 –n vxflexos, the system
indicates that the driver image could not be
loaded.

Edit the daemon.json file found in the registry
location and add
{ "insecure-registries" :
[ "hostname.cloudapp.net:5000" ] }

The kubectl logs vxflexoscontroller-0 –n vxflexos logs shows
that the driver is not authenticated.

Check the username, password, and the
gateway IP address for the VxFlex OS system.

The kubectl logs vxflexoscontroller-0 –n vxflexos logs shows
that the system ID is incorrect.

Use the get_vxflexos_info.sh to find
the correct system ID. Add the system ID to
myvalues.yaml script.

CSI Driver for VxFlex OS 1.0 Product Guide

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.6
Linearized                      : No
Author                          : Dell Inc.
Create Date                     : 2019:04:11 06:42:40-05:00
Keywords                        : "VxFlex OS, CSI, Kubernetes"
Modify Date                     : 2019:04:11 19:38:59+05:30
Language                        : EN
XMP Toolkit                     : Adobe XMP Core 5.6-c016 91.163616, 2018/10/29-16:58:49
Format                          : application/pdf
Creator                         : Dell Inc.
Description                     : This document contains details about CSI Driver for VxFlex OS, including installation instructions, testing information, and troubleshooting.
Title                           : CSI Driver for VxFlex OS Product Guide
Subject                         : VxFlex OS, CSI, Kubernetes
Creator Tool                    : AH XSL Formatter V6.2 MR10 for Windows (x64) : 6.2.12.21390 (2015/07/21 09:32JST)
Metadata Date                   : 2019:04:11 19:38:59+05:30
Producer                        : Antenna House PDF Output Library 6.2.706 (Windows (x64))
Trapped                         : False
Document ID                     : uuid:c6247064-54a5-476b-9249-8ac6afb600cc
Instance ID                     : uuid:d0bfc96e-8949-40b7-8b1e-87237f5bfc8f
Page Layout                     : OneColumn
Page Mode                       : UseOutlines
Page Count                      : 20

EXIF Metadata provided by EXIF.tools

CSI Driver For VxFlex OS Product Guide Vx Flex

Navigation menu

Versions of this User Manual:

Views

Navigation