Technical White Paper Flex Frame Orchestrator – FF14A00

User Manual: Pdf FlexFrame Orchestrator – Technical White Paper FUJITSU Integrated System PRIMEFLEX for HPC - Fujitsu United States

Open the PDF directly: View PDF PDF.
Page Count: 37

WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 1 of 37
ts.fujitsu.com/FlexFrame
FlexFrame Orchestrator
Version 1.4A
THE FLEXFRAME INFRASTRUCTURE SOLUTION - TECHNICAL WHITE PAPER
INTRODUCTION
This document describes the technical aspects of the FlexFrame® infrastructure solution.
The name FlexFrame® is a generic term for both the traditional "FlexFrame® for SAP®" (FF4S) and the current
"FlexFrame® Orchestrator" (FFO).
All technical aspects described in this document and this document itself are subjects of change without further
notice.
NAMING INFORMATION
PRIMEFLEX® is a registered trademark of Fujitsu Limited in Europe and other countries.
PRIMEFLEX for SAP Landscapes enables simplified, fast and secure implementation and operation of SAP
applications and databases. The infrastructure solution is designed, delivered and supported as one product and
supplemented by a broad services portfolio. The integrated FlexFrame Orchestrator software offers consistent
and standardized administration of infrastructure, databases and applications.
TARGET AUDIENCE
The document addresses consultants, administrators and platform decision makers.
We assume that the reader of this document has technical background knowledge in the areas of operating
systems Linux, IP networking and SAP basis.
DOCUMENT HISTORY
Document
Version
Date
WHITE PAPER
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 2 of 37
ts.fujitsu.com/FlexFrame
1.0
2013-05-21
1.1
2014-05-09
1.2
2015-01-16
1.3
2015-03-20
1.4
2015-12-01
1.5
2016-12-05
1.6
2017-12-08
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 3 of 37
ts.fujitsu.com/FlexFrame
Contents
Introduction 1
Naming Information 1
Target Audience 1
Document History 1
1 The FlexFrame® Infrastructure Solution 5
2 Components 6
2.1 Hardware 6
2.2 Software 7
2.3 Shared Operating System 8
2.4 Control Nodes 9
2.5 Application Nodes 9
2.6 Integration of server virtualization into FlexFrame Orchestrator 11
2.7 Role Based Access Control (RBAC) 13
2.8 FlexFrame User Interfaces 14
3 Central Administration via LDAP Configuration Database 14
4 IP Network 15
4.1 Virtual Network Segments 16
4.2 Network High Availability – LAN Failover 17
4.3 Network Switches 17
5 Central Network Attached Storage 18
5.1 Central Network Attached Storage on NetApp FAS Systems 18
5.2 Other NAS types 19
6 Use of SAN storage 19
7 High Availability with FlexFrame Agents 20
7.1 Control Agents 21
7.2 Application Agents 21
7.3 Generic FlexFrame Agent – Autonomy and High Availability for User Specific Services 21
7.4 Spare Application Nodes 21
7.5 Control Node Availability 22
8 Virtualization of SAP Applications 22
8.1 SAP Applications Managed by SAP LVM 22
9 Pools and Groups 22
9.1 Server Pools 23
9.2 Server Groups 23
10 Integration of HANA Services into FlexFrame® 23
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 4 of 37
ts.fujitsu.com/FlexFrame
11 SAP Cloning 24
12 Additional Management Functions 25
12.1 FlexFrame Event Management Expert 25
12.2 Performance and Capacity Management 27
12.3 Accounting Management 27
12.4 FlexFrame Filesystem Utilization and Filespace Accounting Option 29
12.5 Management Reporting Option 30
13 Prerequisites and Assumptions 30
13.1 Computing Center Infrastructure 30
13.2 Host Names and IP Addresses 30
13.3 SAP Service Host Names and IP Addresses 31
13.4 Access Rights 32
14 FlexFrame Backup with Simpana 32
15 Third-Party Software 34
16 Support 34
16.1 Fujitsu Customer Support 34
16.2 SAP 34
17 Benefits of the FlexFrame® Platform 34
17.1 Shared Use of Storage Space 35
17.2 Shared Usage of Servers 35
17.3 Less Administration 35
17.4 Less Downtime during a Change of OS 35
17.5 Improved Change Management 36
17.6 Simplified Multi-Client Operation 36
17.7 No Standby Database Server Required 36
18 FlexFrame® and SAP® Business Suite 37
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 5 of 37
ts.fujitsu.com/FlexFrame
1 The FlexFrame® Infrastructure Solution
The FlexFrame® infrastructure solution is a revolutionary approach to run complex SAP® infrastructure solutions
with higher reliability and efficiency.
At the same time some major changes to the configuration paradigms for infrastructures have been
implemented.
These changes are:
A shared operating system booted over IP networks for the SAP servers.
Decoupling of application software and operating system, called virtualization of SAP software.
Shared Network Attached Storage from NetApp.
FlexFrame Agents providing revolutionary mechanisms to implement high-availability functions without
cluster software.
A new major change started with the first Orchestrator version and is improved step by step:
The solution gets stepwise independent from hardware and software restrictions. That means that some
hardware component can be used by project based on defined interface to be provided by those (e.g. for
network component, NAS storage system or virtual machine support)
FlexFrame Orchestrator manages environments running SAP Enterprise Resource Planning (ERP) applications and
supports consistent and uniform management of all SAP landscapes, including SAP HANA infrastructures. This
solution is designed for all enterprise environments and for every kind of IT provisioning model, regardless of
whether the applications are running in the customer’s data center, delivered as a managed service or a hosting
service from an external provider, or deployed from the cloud.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 6 of 37
ts.fujitsu.com/FlexFrame
2 Components
The concept of FlexFrame consists of several components which implement state-of-the-art functionality.
Together with the FlexFrame Agents, the whole solution is far more than just the sum of its components. A major
part of the benefits consists in a dramatic reduction in day-to-day operating costs for SAP environments.
It is, of course, possible to use parts of the FlexFrame infrastructure solution in individually designed
implementations
1
. In such customer-specific configurations, professional service and maintenance have to be
considered separately.
2.1 Hardware
A typical FlexFrame environment consists of the following types of hardware:
4
1
2
3
Control
Network
Application
FlexFrame
Storage
3
Add-on
1. Control Nodes – the Control Center: Two PRIMERGY servers of type RX300/RX2540 with SUSE Linux Enterprise
Server installed on local disks. Alternatively the Control Nodes can be realized as KVM virtual machines.
1
Such implementations are classified into “standard +” Solution projects, where a standard special release request has to be made to the product owner, and “extended”
(enhancements that later become mainline) and “customized” (enhancements without path into the mainline) Solution projects, where the project manager needs to involve the
product owner in order to get a proper solution, to generate the mutual back-to-back agreement
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 7 of 37
ts.fujitsu.com/FlexFrame
2. Network switches – Cisco® of type Catalyst 3750 or Brocade ICX, and/or Cisco® Nexus or Brocade VDX6740.
3. Network Attached Storage with a NetApp FAS System or a Storage Virtual Machine hosting shared OS file
systems and application data.
4. Intel-based PRIMERGY and PRIMEQUEST Servers (standard rack servers or blade servers) serving as
Application Nodes with SUSE Linux Enterprise Server (Shared OS).
For detailed information about the hardware currently supported in a FlexFrame environment, see the current
FlexFrame Support Matrix (https://partners.ts.fujitsu.com/com/products/infrastruc-solutions/FlexFrame).
Any other functions, such as backup, can be implemented separately as an add-on to FlexFrame and need
dedicated hardware, operating system, high availability, professional service and support etc.
2.2 Software
The FlexFrame infrastructure solution consists of both hardware and software. To enable the complete
environment to function properly, the entire software set is strictly defined. Anything other than the software
components listed below is not part of FlexFrame. This also applies if software from the list below is missing, is
installed in other versions than specified below, or if software other than the current SAP components is added.
No.
Hardware 1
OS
Software
Services
1
Control Nodes:
2 x PRIMERGY RX2540 M1, M2 or M4 or
2 x PRIMERGY RX300 S7, S8, or
2 x PRIMERGY rack servers each equipped with a
KVM hypervisor
SLES12 SP3
FlexFrame Agents (Control Agents),
FlexFrame 1.4A File System Image CN,
ServerView Agents, ServerView RAID
etc.
TFTP, DHCP, LDAP,
(SAPROUTER), etc
2
Network switches:
pairs of 1 GbE switches (Cisco Catalyst 3750x or
Brocade ICX 64xx),
and/or
pairs of 10 GbE switches (Cisco Nexus
50xx/55xx/9236/9372 or Brocade VDX6740 or 3 )
IOS (proprietary)
Brocade Network OS(propr.)
NX-OS (proprietary)
….
(as delivered)
3
Network Attached Storage:
one or more NetApp storage systems
(FASxxxx, AFF80xx, AFFAxxx, vFiler or SVM’s),
disk shelves as required 4
hosting shared OS file systems
and application data
ONTAP (7-mode and cDOT)
(proprietary) 5
NetApp Tools
NFS,
optional:
cluster components,
FlexClone, SnapVault,
SnapMirror, SnapRestore
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 8 of 37
ts.fujitsu.com/FlexFrame
No.
Hardware 1
OS
Software
Services
4
Intel- based PRIMERGY or PRIMEQUEST server
(standard rack or blade server)
SLES11 SP4 / SLES for SAP 2
or
SLES12 SP2 / SLES for SAP 2
or
VMware vSphere ESXi running
SLES11 SP4 / SLES for SAP 2 or
SLES12 SP2 / SLES for SAP 2
or
SLES11 SP4 / SLES for SAP 2
running as KVM Hypervisor,
running SLES11 SP4 / SLES for
SAP 2 or SLES12 SP2 / SLES for
SAP 2
FlexFrame V1.4A File System Image,
FlexFrame Agents (Application
Agents),
SAP Applications, Database
SAP & DB Services
Overview of current software and hardware dependencies in FlexFrame
1 allowed types according to current FlexFrame Support Matrix
2 based on SLES11 SP4 (x86_64) resp. SLES12 SP2 (x86_64)
3 or other switch pairs when adapter modules exist and are certified by Fujitsu
4 The amount of disks required for customer-specific FlexFrame configurations can be determined together with Fujitsu Customer Support Filer Sizing Team
5 The currently supported ONTAP version is kept up-to-date in the FlexFrame Support Matrix
2.3 Shared Operating System
One major aspect of FlexFrame is its shared operating system. Sharing in this case means that the very same files
of essential parts of the underlying operating system are used to run multiple servers. This part of the file system
is mounted read-only, so none of the Application Nodes that run the actual applications can modify it. Server-
specific information is linked to a file system area that is server-specific and mounted read/write. The shared
operating system (normally named “image”) is kept to a NAS system (Network Attached Storage system from
NetApp).
2.3.1 Shared OS Boot Concept
The figure below shows the boot process of a FlexFrame Application Node (PRIMERGY/Linux).
The FlexFrame Concept –Shared OS
PRIMERGY
BX920S4
PRIMERGY
RX4770M2
PRIMERGY
RX900S2
Fujitsu3
Fujitsu4
Fujitsu2
1get boot info
Fujitsu1
2Server Name IP address
where to boot from
3mount root directory
4boot Linux
NAS Storage System
OS-shared
Control Node (clustered)
assigns server names
provides boot info
distributes SAP services
admin/monitoring
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 9 of 37
ts.fujitsu.com/FlexFrame
2.4 Control Nodes
Every FlexFrame environment always includes two Control Nodes. Their purpose is to be a single point of control
for the Application Nodes, as well as to check and manage the proper operation of the Application Nodes.
The two Control Nodes are also called the Control Center.
Control Nodes do not run SAP software (with the exception of saprouter, as an option). They exclusively run SUSE
Linux Enterprise Server Version 12 (based on SLES12 SP3), installed on local disks. Control Nodes provide and run
services such as:
Linux-HA high availability cluster framework
Timeserver for the complete FlexFrame landscape
Control Agents
Web server to provide the Control Agents’ user interface
DHCP for assignment of IP addresses and TFTP for the boot process of the Application Nodes
saprouter (optional)
Control Nodes at present are of the server type PRIMERGY RX2540M1, RX2540M2, RX2540M4 or RX300 Sn, n = 6,
7 or 8. Alternatively they are virtual machines placed on two distinct PRIMERGY rack servers of free choice each
equipped with a KVM hypervisor.
2.5 Application Nodes
Application Nodes are the workhorses of the FlexFrame infrastructure solution.
Application Nodes run database and SAP services on the SUSE Linux Enterprise Server shared operating system.
Application Nodes can be physical servers that offer CPU and memory, or virtual servers built on top of physical
servers, using the physical server's CPU and memory through a virtualization layer.
For FlexFrame Orchestrator 1.4A the principal types of Application Nodes are:
PRIMERGY or PRIMEQUEST server running SLES11 SP4 or SLES12 SP2
Virtual machines running SLES11 SP4 or SLES12 SP2 on a PRIMERGY or PRIMEQUEST server running a
vSphere ESXi or a KVM hypervisor
Admissible servers have to be approved for SAP on Linux by Fujitsu.
During the boot process using Intel's PXE® technology, each Application Node is identified using the hardware
address of its boot interface (MAC address). The Control Node assigns an IP address to it and supplies the
operating system via the network.
Selected file systems (especially the root file system "/") are mounted over the network read-only.
If, for any reason, an Application Node needs to be replaced or added, only a handful of settings need be adjusted
to integrate it into the FlexFrame environment.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 10 of 37
ts.fujitsu.com/FlexFrame
Intel's® PXE technology is implemented in Fujitsu’s PRIMERGY and PRIMEQUEST servers and allows booting via the
network. DHCP is used with static MAC address relationship for all the Application Nodes.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 11 of 37
ts.fujitsu.com/FlexFrame
2.6 Integration of server virtualization into FlexFrame Orchestrator
Virtualization concepts play a fundamental role for the customer value of the dynamic infrastructure FlexFrame
Orchestrator. Besides the concept of virtualization of the SAP software by decoupling application software and
operating system and the concept of virtual IP addresses enabling the switch of services from one host to
another, FlexFrame additionally offers the use of virtualized servers as Applications Nodes.
As of FFO 1.1A00-020 the Control Center of FlexFrame can either consist of physical Control Nodes or of virtual
Control Nodes (vCN). In case of a virtual Control Center (vCC) both virtual machines are hosted on different
physical PRIMERGY servers that run SUSE KVM as hypervisor.
Starting Point is the general virtualization design:
Virtualization Layer
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 12 of 37
ts.fujitsu.com/FlexFrame
Allowing Application Nodes with and without virtual machines the obtained picture in FlexFrame is
FlexFrame makes it possible for an SAP application to be started on any server in the FlexFrame environment. This
can be a physical server or a virtual server. An SAP application can be relocated from a virtual server to a physical
server and vice versa at any time. As a result, the application can be moved to the system with the best suited
resources without any additional installation work. For example, several applications with low loads are run in
parallel on several virtual servers, although if an application’s load is high, a complete physical server is assigned
to it. If this is not sufficient, a server with more or more powerful processors and main memory is chosen.
All in all, the available server capacities can thus be utilized better and thus reduced. Unused resources can also
be switched off. Both measures cut procurement and power costs. Flexible allocation is also a great aid if a new
SAP application does not run correctly on a virtual server. It is then moved to a physical server to test whether the
problem is the result of virtualization or due to other reasons.
Management
All installed and configured resources are controlled from a central instance, the FlexFrame Control Center. It can
essentially be assumed that an SAP environment consists of a mix of virtual and physical servers. Uniform
management of all resources – virtual or physical – is therefore an important requirement for the IT
infrastructure. Central management of the SAP services also offers many advantages in the event of a fault. If a
KVM
physical server as
Application Node
Application Nodes
wearing virtual
machines on a
virtualization layer
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 13 of 37
ts.fujitsu.com/FlexFrame
server has failed, the SAP application is automatically booted on a substitute system. SAP applications are also
restarted on virtual servers. The basis for this is a FlexFrame Agent (see chapter 7) that is started on all application
nodes when the operating system is booted and reports to the Control Center that it has been started and is
running successfully. This agent allows SAP-specific problems to be detected and countermeasures to be initiated.
As a result, not only availability of a virtual or physical server is monitored, but also whether the application is
running smoothly and its utilization. That means administrators have an overview of all server resources and
applications at any time and can also change the assignment of resources to applications. There is thus end-to-
end control of the entire system.
2.7 Role Based Access Control (RBAC)
In computer systems security, role-based access control (RBAC) is an approach to restricting system access to
authorized users. Within an organization, roles are created for various job functions. The permissions to perform
certain operations are assigned to specific roles. Members of staff (or other system users) are assigned particular
roles, and through those role assignments acquire the permissions to perform particular system functions.
In order to configure and administrate a FlexFrame environment numerous programs have to be used, nearly all
of them needing root permissions to do their job. So the previous FlexFrame versions for example did not allow to
separate administration for different pools (and often a pool corresponds to a customer in outsourcing
environments)!
With RBAC a FlexFrame administrator
need not have “root” privilege any more
can be a named account on the Control Center
can be restricted for different administration scopes (pool, sid, ..)
will be logged with all its interface actions done by FlexFrame programs
RBAC uses two data structures to define access permissions:
roles
A role contains all permitted FlexFrame programs and operation modes for this role.
user roles
A user role maps FlexFrame administrator roles by pool to a user role name. This name has to match to an
account or group name. If the account name matches any user role no more user roles are scanned. If no
user role name matches the account name all the groups of the account are used to scan for user roles.
The permissions of the group name matching user roles are added to a composed virtual user role.
2.7.1 RBAC standard role definition
Out of the box FlexFrame comes with six role definitions. Three of them are for configuration/administration and
the other three for observation. They are divided into SID (SAP system identification) specific, pool specific and
main FlexFrame and do not overlapping in most cases. This means to have grants to SID and pool specific tools
both roles have to be added to user role. To get a non root user with root like FlexFrame permissions all three
configuration/administration roles have to be added to the user role of the user.
The three configuration/administration roles are:
ffadmin
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 14 of 37
ts.fujitsu.com/FlexFrame
pooladmin
sidadmin
The three observer roles are:
ffobserver
poolobserver
sidobserver
The intention of the predefined rules is to give a non root user the ability to do all the SID or pool specific
FlexFrame jobs or to do FlexFrame environment related work. These are divided into a “read only” observer part
and an administrative part.
2.8 FlexFrame User Interfaces
A FlexFrame landscape can be accessed through Secure Shell (ssh) connections to the Control Node. Any other
remote administration tools like rsh or telnet have been disabled for reasons of security.
The FlexFrame Control Nodes provide a web portal with links to Web interfaces of several FlexFrame
components, i.e. FlexFrame Agents WebGUI (see chapter 7 about the FlexFrame Agents) and the FlexFrame
documentation.
Beyond that the administration of FlexFrame is performed via CLI and GUI. The FlexFrame Administration GUI is
offered additively for most of the standard administration tasks. This GUI is realized as a part of the FlexFrame
Management Tool, which in Planning Mode enables the certified FlexFrame consultant to enter the necessary
configuration data for a customer-specific FlexFrame environment. And in Administration Mode provides an easy
and comfortable way to administrate the FlexFrame system environment. In Administration Mode the CLI
commands are generated automatically by the Management Tool and need not be typed in manually.
3 Central Administration via LDAP Configuration Database
LDAP is used as the central information service for all shared OS nodes within a FlexFrame environment. The
Control Nodes are used as LDAP servers. The LDAP database is located on shared file systems mounted from the
NAS storage. The Application Nodes are configured as LDAP clients. LDAP requests from Application Nodes are
restricted to the data of their own pool.
LDAP provides host-related network information such as:
net boot
automount
user authentication
groups
host names and IP addresses
shared services
networks and netmasks
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 15 of 37
ts.fujitsu.com/FlexFrame
LDAP client profiles
Additional information about configuration data is only applicable for Control Nodes. These are used FlexFrame
internally to add, remove or modify the configuration of Application Nodes or SAP services.
The standard high availability and failover solution “Linux-HA cluster” is used for the two Control Nodes. This
feature also guarantees that after breakdown of a Control Node the cluster provides its IP address and the
corresponding LDAP resource (LDAP-master or LDAP-replica) to the other node and makes it available again for
the entire system.
4 IP Network
The network is the backbone of the FlexFrame infrastructure solution. Communication between the various
nodes and storage devices is done exclusively via the IP network infrastructure. This is used both for
communication between server(s) and client(s) and for delivering IO data blocks from the NAS (Network Attached
Storage) to the server.
The IP network infrastructure is essential for every FlexFrame configuration. FlexFrame is designed to use
different IP networks for different purposes.
The following networks are mandatory:
client
The purpose of the client network is to have dedicated user connectivity to the SAP instances. This network
also allows administrators to access the Control Nodes. This network can be routed outside the FlexFrame
networks to connect to the existing network.
control
The control network carries all FlexFrame global administrative communication, e.g. between control nodes
and iRMC or management blades or to change switch or NAS configuration or similar purposes.
server
The server network is used for the communication between SAP instances among each other and the
databases.
storage
The storage network is dedicated to NFS communication for accessing the Application Nodes' shared
operating systems, the executables of SAP and the RDBMS as well as the IO of the database content and SAP
instances.
The following networks are optional:
admin
The admin network can be added to have dedicated user connectivity for server administration. If missing the
client network must be used.
backup
The backup network can be added to separate backup traffic of an application node and a backup server. If
missing the server network can be used.
hanaint
The hanaint network is dedicated to HANA scale out solutions to carry the HANA internal traffic.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 16 of 37
ts.fujitsu.com/FlexFrame
hanarep
The hanarep network is dedicated to HANA system replication.
sapdata
The sapdata network can be added to separate NFS communication for accessing database files. If missing the
storage network is used.
saplog
The saplog network can be added to separate NFS communication for accessing database log files. If missing
the storage network is used.
livemig
The livemig network can be added to separate communication for virtual machine live migration. If missing
the control network can be used.
4.1 Virtual Network Segments
The FlexFrame network concept relies on virtual LAN (VLAN) technology that allows running multiple virtual
networks across a single physical network. Every FlexFrame IP network is associated to a dedicated VLAN,
reserved for FlexFrame traffic only.
The following figure outlines the basic virtual network segments of a typical FlexFrame landscape with Application
Nodes.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 17 of 37
ts.fujitsu.com/FlexFrame
The bandwidth of the physical network has to be at least 1 Gbit/s for all components to handle intense data
traffic. Selective network connections (for instance the connections from a FlexFrame NetApp NAS system or from
a set of 10GbE-capable rack servers or blade servers (via switch blades) to the FlexFrame access layer switches)
may be chosen as 10Gbit/s connections.
4.2 Network High Availability – LAN Failover
In FlexFrame every network runs through a logical network interface that consists of several physical network
ports which in turn are using redundant network paths (cables and switches). This includes a double switch and at
least double wiring infrastructure. A single logical redundant interface may be sufficient to get access to every
network needed.
When a network component (port, cable, switch, etc.) fails, the network management logic will switch over to
another network port and path. This will be described as „LAN failover“.
4.3 Network Switches
Network switching components play a very important role within FlexFrame. Currently the following switches are
tested and supported as FlexFrame access layer switch:
Cisco Catalyst 3750 series
Cisco Nexus series
Brocade ICX 6430 / 6450
Brocade VDX 6740
These switch families support VLAN technology for flexible configuration of the various network segments and
cross-switch link-aggregation. For cross-switch link-aggregation the Cisco Catalyst models use the Cisco
StackWise® Technology and the Brocade ICX models an analogue stacking technology, the Cisco Nexus models
build a vPC domin and the Brocade VDX models a VCS fabric. Cross-switch link-aggregation is currently used for
NAS system connections, switch group interconnects and switch blade connections.
There are two different network speeds supported:
1Gbit/s
10Gbit/s
With the 3750G and the Brocade ICX 6430 models you can only use 1Gbit/s network speed. With the 3750E or
3750X, the Nexus, the Brocade ICX 6450 and Brocade VDX 6740 models you can use 1Gbit/s and 10Gbit/s network
speed.
For 1Gbit/s you can use copper or fiber ports. For 10Gbit/s you normally use fiber ports.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 18 of 37
ts.fujitsu.com/FlexFrame
The 10Gbit/s network speed can be used for the connection of 10GbE-capable NetApp FAS Systems as well as for
the connection of 10GbE-capable rack servers and blade servers and as uplink to the customer core network.
The 1Gbit/s network speed is used for the connection of the Control Nodes and management ports.
As of FlexFrame Orchestrator 1.2A there is a network API defined allowing the introduction and support of new
switches in FlexFrame solely by
an XML description
a description for how to do an initial refueling
a switch type specifically realization (of the API) by a Perl adapter-module
The adapter-module has to be certified by Fujitsu.
5 Central Network Attached Storage
The storage for all Application Nodes is consolidated on one (or more) central Network Attached Storage systems
with a connection based on the network file system protocol (NFS). The central NAS storage is a FAS System from
NetApp or a NetApp FAS system gateway connected to a Fujitsu ETERNUS storage system of type DX.
The NetApp implementation of the NFS (Networked File System) allows the same data files to be shared by
multiple hosts and thus provides a built-in cluster file system.
The FlexFrame concept reduces the amount of "lost" disk space, since multiple SAP systems can optionally share
the same set of disks. As the data grows, it is easy to add additional disks and enlarge the volumes without any
downtime.
The storage configuration for FlexFrame concerning capacity and IO requirements are determined by the Fujitsu
SAP Presales process.
The FlexFrame solution comprising a pool of physical and virtual servers, a central storage system, an intelligent
control instance (Control Center) and a network linking all components together is a self-contained system. As
one implication of this paradigm it is not allowed to share the central storage system with applications from
outside of FlexFrame.
5.1 Central Network Attached Storage on NetApp FAS Systems
The operating system of the Filer is called "ONTAP". The disks will be used to build an aggregate. FlexVolumes
can be created on top of aggregates. A FlexVolume contains a file system and is usable from NFS (UNIX) or CIFS
(Windows) client systems. Data is stored using a Write Anywhere File Layout (WAFL) mechanism.The Filer has
Non Volatile RAM (NVRAM) that buffers committed IO blocks. The contents of the NVRAM will remain intact if the
power of the Filer should fail. Data will be flushed to the disks once power is back online.
The minimum FlexFrame environment has at least the following volumes:
volFF (OS images of Application Nodes, SAP and database software, pool related files)
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 19 of 37
ts.fujitsu.com/FlexFrame
sapdata (database files)
saplog (database log files)
If the NAS system is a Filer from Network Appliance using the conventional ONTAP 7-mode the volume vol0
(ONTAP, configuration of Filer) is necessary too.
The amount of disks required for customer-specific FlexFrame configurations can be determined together with
Fujitsu’s Global Competence Centre Architecture Services Team.
Together with ONTAP 8.0 NetApp introduced its new paradigma of a Storage Virtual Machine (SVM) with its
“Clustered Data ONTAP (cDOT)”, which is on the way to replace the conventional ONTAP 7-mode, and which is
the basis for NetApp’s scale-out architecture.
As of FlexFrame Orchestrator 1.1A Clustered Data ONTAP is supported and the 7-mode DATA ONTAP support is
continued as well.
A NetApp storage system’s or a Data ONTAP cluster’s networking and storage resources can be partitioned either
with the help of Data ONTAP MultiStore technology into virtual filers or resp. with clustered Data ONTAP’s
technology into SVMs (storage virtual machines). Each virtual filer (vFiler) resp. SVM logically appears on the
network as a separate storage resource and is dynamically configurable. Along the lines of FAS systems also V-
series devices, vFilers and SVMs are supported within FlexFrame Orchestrator.
Usage of multiple Filer as well as NetAPP HA configurations (in contrast and additively to cDOT cluster
configurations) is possible for redundancy and reliability. A storage system can be clustered to protect data
against the failure of a single FAS System by specific NetApp tools. This switching from one storage system to its
HA or cluster counterpart is transparent to the FlexFrame Application Nodes.
5.1.1 Snapshots
When a snapshot is taken, no data blocks are copied; just the information where the data blocks are located is
saved. If a data block is modified, it is written to a new location, while the content of the original data block is
preserved. A snapshot can therefore be created very quickly since only a small amount of data need be copied.
The use of snapshots does not decrease the throughput and performance of the storage system.
Snapshot functionality allows the administrator to create up to 250 backup-views of a volume.
5.2 Other NAS types
FlexFrame Orchestrator defines an API for NAS. Providing an implementation of the NAS API allows the support of
different NAS systems. Realizing a new NAS API is not covered by the FlexFrame Orchestrator core functionality
and must be approved in a SFG process (Special Release Requests).
6 Use of SAN storage
If the requirements to the IO-performance of the used SAP systems and SAP services are extremely high, there is
an alternative way to the use of NAS storage for the productive SAP databases: as of FlexFrame Orchestrator 1.1A
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 20 of 37
ts.fujitsu.com/FlexFrame
instead of using volumes from a NAS system over NFS, the database datafiles (sapdata) and database logfiles
(saplog) of a SAP system can be placed on SAN based storage.
Configuring the SAN storage environment for a FlexFrame landscape is a complex task that needs careful planning
and design. A team of specialists for storage systems, servers and SAN infrastructure must work out a concept
that will be a guideline for the complete setup.
The SAN configuration for FlexFrame will usually be done by a storage systems expert in cooperation with a
member of the FlexFrame implementation team.
The used SAN storage system and Fibre Channel switches are not considered as a part of the FlexFrame system;
you can use LUNs from a storage system which is also used for other purposes in your computing environment.
Also the administration and configuration of the SAN fabric and the Fibre Channel switches is completely in the
responsibility of the customer. The FlexFrame software itself does not need an administrative access to the SAN
storage system nor to the FC switches.
To manage the multiple paths between host and storage system, a multipath software is needed.The FlexFrame
Linux Application Node Images contain a ready-to-use installation of the Linux native multipath software DM-
MPIO.
And last not least all Application Nodes that will run database instances of SAP systems configured for SAN usage
must be equipped with Host Bus Adapters (HBAs).
All Application Nodes on which a database instance will run must be able to access the LUNs used by the
database. This are the Application Nodes of the poolgroup to which the Node where the instance is first started
manually belongs.
7 High Availability with FlexFrame Agents
The following sections describe how SAP and database services are made highly available using FlexFrame Agents.
The FlexFrame Agents consist mainly of two different types of component, the control agents and the application
agents. These components together monitor hardware and software availability, and can intervene if any
component of the FlexFrame environment should not react properly.
This intervention feature is a key feature of the FlexFrame infrastructure solution. Seen from a client viewpoint, it
provides automatic repair of almost any imaginable failure.
The main intervention mechanisms used are
Stop and restart services
Reboot node and restart services
Switch over services to another node and shut down/reboot the former host
Power-off a node (in case of damage)
Displace services of lower priority for services of higher priority if needed
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 21 of 37
ts.fujitsu.com/FlexFrame
Start a service on a node in addition to another service running there
7.1 Control Agents
The control agents run on the Control Nodes and basically monitor the application agents. If an application agent
is no longer responding, the control agent has to react and restart the application agent's services on a spare
Application Node. To avoid the split brain error, the control agent initiates a power-off sequence for the
Application Nodes that failed. This power-off sequence is issued via system-specific power-off hardware and
software facilities.
7.2 Application Agents
While an Application Node is booting, the autonomous application agents are started. Once an application (SAP
instance, database etc.) is started, the application agents will start monitoring it. No extra configuration is
required. If an application should fail or cannot be started, the agent will try to re-start it, and if this fails, the
entire service is started on a different Application Node (spare node). Agents on spare nodes compete for such
failed services and the "winner" starts the service. Failed services can also be started on active Application Nodes,
for this purpose there exist sophisticated possibilities of add rules, replace rules and substitution rules matching
customer specific circumstances.
7.3 Generic FlexFrame Agent – Autonomy and High Availability for User Specific Services
In many user scenarios additional SAP or non-SAP services shall be utilized on the application servers in the
FlexFrame environment, so there is a need to integrate these services into the FlexFrame autonomy rules.
With the FlexFrame Agents it is possible to integrate other services into the autonomy scenarios. The functionality
allows pool specific definition of autonomy rules for additional services.
In this case the rules for detection, reaction and high availability have to be defined in rule files. The generic
FlexFrame Agent functionality allows an adaptation to project specific requirements. For this purpose a generic
service is defined through a set of parameters which are used for its identification and which generate the service
states. The description and definition of a service is arranged in state, detection and reaction models.
7.4 Spare Application Nodes
Legacy concepts (non-FlexFrame) have dedicated clusters for fixed services. In those concepts a complete set of
redundant failover hardware is required, which is idle as long as no error occurs.
In FlexFrame any number of spare nodes will stand-in if any Application Node fails.
In FlexFrame the concept of server pools and groups is new invented. In a server group, only such types of server
are configured which can stand-in for each other. Depending on the number of SAP services and their requested
SAPS power, a number of servers may run in a server group (see also chapter “Pools and Groups“).
Spare servers and servers running services of low priority are the failback resources to enable any SAP or
database service that may fail to move to another node in the same server group.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 22 of 37
ts.fujitsu.com/FlexFrame
Spare server could be individually allocated to a pool and could stand-in for any failure of an Application Node
(with same OS image) in that pool. Additionally pool-independent spare servers are possible, which are
assembled in a special spare Pool named Adminpool, and which can stand-in for any failure of an Application
Node in any pool.
7.5 Control Node Availability
Control Nodes monitor and control the whole FlexFrame environment. Therefore Control Node functionality itself
has to be highly available. The standard high availability and failover solution “Linux-HA” is used for the two
Control Nodes. Linux-HA is an integrated part of SUSE Linux Enterprise Server supported in FlexFrame.
8 Virtualization of SAP Applications
Before the concept of FlexFrame was created, it was common practice to install SAP and database services on a
"fixed" server.
Usually one IP address was used for:
the host itself
database
each SAP Instance
Once installed, the service could not easily "be moved away" from its host.
In a cluster configuration, virtual IP addresses were introduced to be able to switch a service from one host to
another. FlexFrame is using this technology for virtualization. This concept is one of the fundamental pillars
enabling FlexFrame to constitute as a Dynamic Infrastructure Solution.
8.1 SAP Applications Managed by SAP LVM
With the integration of SAP LVM (Landscape Virtualization Management) Standard Edition, formerly SAP ACC
(Adaptive Computing Controller), FlexFrame provides the standardized virtualization layer based on SAP
NetWeaver™, this means the virtualization of released SAP services and the use of SAP LVM to manage SAP
services with additional benefits (task planner & mass operation e.g. start or stop a complete landscape).
The LVM provides a "userexit" which is used to inform the FlexFrame Agents. In this way the LVM is integrated
into the environment of Control and Application Agents. While the LVM provides a central console for the SAP
instances, the FlexFrame Agents keep services and nodes highly available and execute self-repair functions.
The use of SAP LVM is an optional FlexFrame feature.
9 Pools and Groups
FlexFrame offers advanced functions for partitioning a FlexFrame environment into service-specific or customer-
specific server pools and groups. This may be interesting for large installations or application service providers.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 23 of 37
ts.fujitsu.com/FlexFrame
9.1 Server Pools
A pool is a number of Application Nodes belonging to the same department or a customer with exclusive
hardware requirements. FlexFrame systems can be divided into pools. Each FlexFrame system consists of at least
one pool. In a pool, all servers may communicate with each other, but not with the servers of other pools.
Servers of different pools can use different copies of the OS.
Server Pools are separated into different network segments. Server LAN, Storage LAN and Client LAN are pool
specific.
The Control LAN is connecting the Control Nodes with all servers of all server pools.
All server pools share the Control Nodes, the Control LAN segment and the NAS storage containing the common
volFF volume.
9.2 Server Groups
Within a server pool, various types of hardware can be used with different characteristics, such as operating
system, architecture, number of CPUs and RAM size. The bulk of servers can be divided into groups of servers
with similar operating systems and hardware performance. Servers in a single group have to be able to take over
the services of each other server in this group. This ability may be very useful for groups of high-performance
database servers (e.g. ORACLE or HANA database servers, see chapter 10) or groups of medium-performance
application servers. Each pool consists of at least one group. In FlexFrame groups are denoted by server groups or
pool groups.
Each SAP application running in a pool can use one or more servers in one or more groups of servers in the same
pool. Each instance of this application runs in a selected group. In case of a failure, switchover to another server is
possible in the same group. For example, a set of servers can be divided into high-performance database servers
and smaller application servers. The group configuration makes sure that database instances run in a group of
high-performance servers, while application instances stay on groups of smaller servers without interfering with
each other.
All servers in a pool share the same LAN segments, even if they belong to different groups. They do not share LAN
segments with other pools, except the Control LAN.
10 Integration of HANA Services into FlexFrame®
FlexFrame Orchestrator is the solution to manage all SAP infrastructure landscapes including HANA. Besides the
offerings of Fujitsu infrastructure solutions for PRIMEFLEX for SAP HANA appliances stands the HANA integration
into the FlexFrame Orchestrator solution.
In a first step the so-called HANA Single Nodes (where the HANA DB or HANA service runs on one single (physical)
host) have been integrated in FlexFrame Orchestrator 1.0A - you can define a HANA group in any pool, where this
HANA group consists of servers “suitable” for HANA applications. Thus analogously to the hitherto supported
SAP databases SAPDB, ORACLE, SYBASE and DB2 also HANA “in memory” databases installed on a FlexFrame
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 24 of 37
ts.fujitsu.com/FlexFrame
application node are supported with the full FlexFrame range of functions as well as HANA services are supported
similar to other SAP services like e.g. BOBJ, SMD or TREX.
The FlexFrame highlights like “any service on any server at any time”, as are the virtualization of SAP services, the
“automatic repair of almost any imaginable failure”, the shared use of servers and storage space, the less
downtime and improved change management (see chapter 17) are thus also valid for any HANA Single Node
system within FlexFrame Orchestrator. The difference regarding automatic repair failure is:
If HANA System Replication is used IT IS NOT DONE by FlexFrame agent functionality. That means no automatic
takeover from the primary to the secondary site if the primary as a whole fails.
From version FlexFrame Orchestrator V1.2 on you can also define a HANA scale out (or HANA Multi Node)
configuration within FlexFrame Orchestrator. This specific configuration is configured and deployed with the same
management tasks as for other SAP application nodes (means e.g.: single image concept, integrated network,
shared storage). The different regarding automatic repair of failure is:
IT IS NOT DONE by FlexFrame agent functionality but only by the inherent high availability functionality of SAP
HANA itself.
That means also that within FlexFrame Orchestrator the monitoring of the state of HANA server and HANA
services is done by the FlexFrame agent. But a failure reaction is done by HANA high availability tasks only as that
is the SAP supported functionality.
11 SAP Cloning
FlexFrame offers a SAP cloning function which performs an automatical cloning of an entire SAP installation from
one pool to another. With this function also the binaries are copied to the target pool and so you get a complete
independent copy of your source system! The command does most configuration tasks needed to run the SAP
system in the target pool.
The cloning can be done even if the system is active and the database is online. As a storage-based snapshot is
prerequisite, restrictions in place on snapshot based backup/recovery for SAP HANA apply.
(Starting with HANA 2 SPS 01 storage snapshot support is available for multitenant database containers that
consist of one tenant database only. On details see https://launchpad.support.sap.com/#/notes/2039883 .)
The cloning cannot be done for HANA Scaleout configurations.
This cloning function is started with a CLI-command and is performed in a time interval of only a few minutes: the
central component is NetApp’s feature FlexClone allowing to make very fast and space-efficient copies of so-
called FlexVol® database volumes. The feature requires an extra license key.
The user is responsible for license issues using SAP software, as this feature copies a complete SAP system.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 25 of 37
ts.fujitsu.com/FlexFrame
12 Additional Management Functions
The following section describes additional functions of management options which help to fulfill requirements
such as
FlexFrame event and fault management expert
FlexFrame performance and capacity management
FlexFrame accounting management
FlexFrame reporting option
These functions are generally useable within FlexFrame. They are based on the raw information which is provided
by the FlexFrame Agents.
For detailed information on this chapter see also the FlexFrame manual
FlexFrame Agents Installation and Administration
12.1 FlexFrame Event Management Expert
To view the trap in the virtual, physical and service context FlexFrame provides
standardized traps with events
pool and group information
the virtual and physical server and service information
The events are based on the FlexFrame Messenger service and part of every FlexFrame installation. The
standardized FlexFrame trap format and the FlexFrame MIB for the trap format make it easy to integrate in every
management platform.
In addition filtering and routing of traps is possible. The FlexFrame Messenger provides a standard MIB structure,
this is the base for the fast and easy integration in Enterprise IT Management products, one of the additional
values is the frame id to identify FlexFrame Events very easy in an enterprise IT console.
The event management option consists of log file agent to produce SNMP traps from log files e.g. the SAP CCMS
log files and to integrate it into the FlexFrame IT management environment with a central IT Data Warehouse
(Repository, ITDW).
For detailed information on this chapter see also the FlexFrame manuals
FlexFrame Messenger Concept and Usage
FlexFrame LogAgent Concept and Usage
FlexFrame Agents Accounting
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 26 of 37
ts.fujitsu.com/FlexFrame
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 27 of 37
ts.fujitsu.com/FlexFrame
12.2 Performance and Capacity Management
The FlexFrame Agents provide optional performance and accounting data. The agents collect node-based, service-
based and group-based information. The performance management meets the need for monitoring and historical
trending and support intelligent orchestration of resources.
The FlexFrame performance and accounting option requires the activation of additional services on the Control
Node. This service does a performance and accounting calculation of the raw data.
The following graphic shows the architecture of the performance and accounting option.
1
FlexFrame® Performance and Accounting
FlexFrame
Application-
servers
FlexFrame
Database-
servers
Network
Storage
DB CI APP
FlexFrame-Application
Agents
FlexFrame- Performance
and Accounting Service
ITDW
Accounting and
Performance Collets
12.3 Accounting Management
The accounting option is, like the performance option, an optionally activated part of FlexFrame. The production
of the accounting data is a multistage process determining accounting data through aggregation and analysis of
the recorded raw data. The accounting option is available for performance values of the server and services and
also for the file utilization accounting.
The performance option enables monitoring and evaluation of the server and services over a longer period of
time. For every node the following data are available as a minimal, average and maximum value:
load of SAP, database or generic services
other services
Machine idle
The data of the performance and accounting option can be directly visualized with the FlexFrame Agents WebGUI
with performance and accounting management plug in. The granularity of the view and the time span can be
freely defined.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 28 of 37
ts.fujitsu.com/FlexFrame
Important parameters for the accounting are detection and report cycles. The detection cycle defines the number
of measurements within a report cycle. The minimum, maximum and average values are calculated on the basis
of individual measurements for a report cycle. The detection cycle therefore always corresponds to the detection
cycle of the FlexFrame Agents, which is also a parameter for the autonomy function.
In addition to the accounting data the FlexFrame IT management option allows you to transform your accounting
data directly to billing data. A flexible transformation table allows you compute directly the billing value. The
billing factor can change independency from service contract number, system ID, service type and day or time
ranges. So you get a flexible out of the box billing model absolutely integrated in the FlexFrame environment with
no extra effort.
12.3.1 Filesystem Monitoring
Another feature together of the FlexFrame Agents is the filesystem monitoring. The FlexFrame Agents are able to
monitor the filesystem utilization for configured filesystems. For each filesystem individual thresholds can be
defined. On transgression of the limits an event is generated and can be dispatched with the FlexFrame
Messenger expert functionality.
12.3.2 Performance, Capacity and Accounting Profiles
The Performance, Capacity and Accounting collects statistics on CPU and SAPs consumption calculated on server,
system or service level to provide powerful analysis and reporting capabilities. An additional feature is to define
profiles in which limiting values for the CPU and SAPs consumption can be defined. On transgression of the
limiting values a FA performance event is generated.
The profiles can be defined on servicelevel, systemlevel and nodelevel. The user can individually decide which
profiles exist and for which values events have to be generated. A profile can apply to all systems, nodes and
services or pool-, system- , service-, group- or node-specific, depending on the hierarchylevel of the profile.
CPU/ SAPS Values
0.15€
0.20€
0.30€
0.15€
0.25 €
Accounting
Price
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Service
level rule
SC_12345
SC_12345
SC_12345
SC_12345
SC_12345
Service-
Contract
Standard
Standard
Standard
Standard
Standard
Accounting
rule
24:0000:00daily01.01.210001.01.2006DBP23
24:0000:00daily01.01.210001.01.2006otherP23
24:0000:00daily01.01.210001.01.2006allQ22
P22
P22
System
ID
all
all
Service
typ
00:00
00:00
from
Time
24:00
24:00
to
Time
weekend01.01.210001.01.1900
workday
01.01.210001.01.1900
UnitdayType
toDatefromDate
0.15€
0.20€
0.30€
0.15€
0.25 €
Accounting
Price
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Sapsrule
Service
level rule
SC_12345
SC_12345
SC_12345
SC_12345
SC_12345
Service-
Contract
Standard
Standard
Standard
Standard
Standard
Accounting
rule
24:0000:00daily01.01.210001.01.2006DBP23
24:0000:00daily01.01.210001.01.2006otherP23
24:0000:00daily01.01.210001.01.2006allQ22
P22
P22
System
ID
all
all
Service
typ
00:00
00:00
from
Time
24:00
24:00
to
Time
weekend01.01.210001.01.1900
workday
01.01.210001.01.1900
UnitdayType
toDatefromDate
Aggregation cycle
Accounting cycle Accounting report
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 29 of 37
ts.fujitsu.com/FlexFrame
12.3.3 Dynamic Workload Management
The Performance profile together with the FlexFrame Messenger can be used to establish a dynamic workload
management. The average CPU load event can activate a reaction on the FlexFrame Messenger to start an
additional application server.
12.4 FlexFrame Filesystem Utilization and Filespace Accounting Option
This option enables the observation of filesystems and the node- and system- or service-based accounting of the
filespaces. The fill degree of filesystems on local servers and also on storage systems can be supervised through
the filespace accounting system. On reaching a specific percentual threshold this system will generate an event.
The second feature of the filespace accounting option is to store values for the filespace accounting in the IT
Datawarehouse repository.
The following graphic shows the architecture of the filesystem utilization:
1
FlexFrame® FlexFrame File utilization Management
FlexFrame
Application-
server
FlexFrame
Database-
server
Network
Storage
DB
Service
Node
FlexFrame
App Agents
FlexFrame-
Domain
Manager
ITDW
File utilisation Collections
FlexFrame-FrameAgent
Vol FF
Vol 0
System
Group
Frame
Group
The data for filesystem utilization and filespace accounting can be determined with the FA-App-Agents and the
FA-Frame-agents. The parameter- and rulesets for the filesystem utilization allow very granular definitions, in
which cycles and for which filesystem or file the storage consumption and the increase will be ascertained.
Statistic definition in the ruleset
The observed filesystems, directories and filedata can be staticly defined in the ruleset, i.e. the start of the
FlexFrame Agents activates the collection of the data and the calculation for every reportcycle.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 30 of 37
ts.fujitsu.com/FlexFrame
Dynamic definition in the ruleset
The ascertainment of the filesystems, directories and filedata can also be achieved in dependency to systems
and/or specific servicetypes. The generation of the filesystem utilization data ensures only if the defined system
or a specific servicetype actually runs on a node. The FA-App-Agents ascertain through the dynamic triggering.
12.5 Management Reporting Option
Based on a crystal reports technology with integrated viewer, predefined report templates can be used directly to
report event, performance and accounting data.
13 Prerequisites and Assumptions
Certain global prerequisites and assumptions apply for the FlexFrame concept, and these have to be taken into
consideration.
13.1 Computing Center Infrastructure
FlexFrame is based on an existing computing center infrastructure. A stable air conditioning is expected. The
necessary cooling capacity is depending on HW configuration.
An uninterruptible electric power supply and at least for the Control Nodes two independent current entries are
highly recommended.
13.2 Host Names and IP Addresses
Host names and their correspondent IP addresses are used by FlexFrame system software for addressing objects
over a selected LAN segment. The following table gives details about the different networks. The listed suffix is
the suffix appended to a base name to build a host name for an IP address. The base name is the node name in
case of a Control Node, Application Node, Hypervisor Node or NAS system or derived from the SID and instance
number in case of a SAP service as described below.
Networks
network
scope
suffix
associated to
control
global
-co
CN, iRMC, MMB, Switch, HN, NAS
livemig
global
-lm
HN
admin
pool
-adm
CN, AN
backup
pool
-bak
AN
client
pool
CN, AN
hanaint
pool
-hni
HANA interconnect
hanarep
pool
-hnr
HANA System Replication
sapdata
pool
-sd
CN, AN, NAS
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 31 of 37
ts.fujitsu.com/FlexFrame
saplog
pool
-sl
CN, AN, NAS
server
pool
-se
CN, AN
storage
pool
-st
CN, AN, NAS
13.3 SAP Service Host Names and IP Addresses
The SAP service host name and its corresponding IP address are used by applications for addressing the SAP
service. The IP address will be assigned to a host interface before an SAP instance is started. If a SAP service is
moved to another node this SAP service is again addressable using the same IP address.
SAP service host names or its corresponding IP addresses are also used for communication from applications
outside FlexFrame systems, i.e. SAP front ends such as SAPGUI.
SAP service host names are generated automatically by the FlexFrame Management Tool, that supports the
planning and configuration of a FlexFrame environment.
Formation rules for SAP service host names
The SAP service host name is formed from the service type, the two-digit instance number for the service type
app, the SID and the network suffix as described above:
<service_type>[<ID>]<SID>[<network suffix>]
<service_type> can be one of:
db - database instance
hdb - HANA database instance, Single Node Configuration
hdb-mn - HANA database instance, Scaleout Configuration
ci - central instance (ABAP)
pai - primary application instance (ABAP), replaces ci as of SAP 7.4
app - application instance (ABAP)
scs - SAP central (common) services (JAVA)
ascs - SAP central (common) services (ABAP)
jc - JAVA central instance
j - JAVA application instance
lc - Live cache instance
ers - Enqueue Replicated Server instance
bobj - SAP business objects (SPOB)
cms - Content Server (CMS)
mds, - Master Data Management Services (MDM)
mdss,
mdis
smd - Solution Manager Diagnostic Agent
trx - Search and Classification (text retrieval and information extraction)
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 32 of 37
ts.fujitsu.com/FlexFrame
wd - SAP WebDispatcher
<ID> is a number from 00 to 96 (except 2, 25, 43, 72, 89) for app and j only. It is empty for other service types.
<SID> is the system ID of a SAP system.
This host name formation rule is mandatory for the FlexFrame infrastructure solution. Some components rely on
this rule.
In a FlexFrame environment, each node name must be unique. However, each node may have multiple host
names that are derived from the node name using a defined naming rule.
In the SAP environment, host names are currently limited to 13 alphanumeric characters including the hyphen (“-
“). The first character must be a letter. In the SAP environment host names are case-sensitive (see SAP Note No.
611361).
13.4 Access Rights
The current implementation of the NFS client does not allow specification of the client's IP address; hence the
virtual addresses cannot be used for NFS exports (which control access to data paths). Therefore any client in a
specific pool can and must be able to mount any path of this pool.
This condition is valid within each server pool but is not permitted between different server pools.
14 FlexFrame Backup with Simpana
A common concept for backup & restore for all types of databases in a FlexFrame environment has been set up
with CommVault Simpana software.
Our backup concept of the database backup is performed using Simpana’s IntelliSnap feature together with
Simpana’s database agent. It brings the database in backup mode and takes a hardware snapshot on the NetApp
system.
The Simpana software including the UNIX File System iDataAgent, MediaAgent and the special database
iDataAgent (one for each type of database, e.g. MaxDB, Oracle, DB2, SYBASE, HANA) are required to be installed
on the application nodes. All SIDs need to be configured as virtual database instances in Simpana. All Simpana
clients are configured in the Server LAN IP address range and the backup data traffic is performed in the Storage
LAN.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 33 of 37
ts.fujitsu.com/FlexFrame
0© [YEAR OF CREATION, e.g. 2013] [LEGAL ENTITY]
INTERNAL USE ONLY
ETERNUS CS8000
Tape Library
ETERNUS LT
Proxy
Simpana
MediaAgent
Management Server
Simpana
CommServe
Snapshot-
Backup
Backup selected
snapshot
Backup infrastructure can
be additionally used to
backup other applications
and file data.
Mounts
snapshots
Management Server
FlexFrame
Control Node
FlexFrame Application Node 1
Production Server
SAP
Application
FlexFrame Application Node 2
FlexFrame Application Node n
data
data
data
Snapshot 1
Snapshot 2
Snapshot 3
Snapshot 4
NetApp NAS Filer
Triggers and manages
snapshots plus
quiesces application
The backup is performed using Simpana’s IntelliSnap feature in combination with Simpana’s database agent
(iDataAgent). They bring the database in backup mode and take a hardware snapshot on the NetApp system. In
order to meet longer term backup requirements the snapshot will be mounted on the so-called Simpana Proxy
System, a server running a Simpana MediaAgent instance and having access to secondary storage like tape. Using
NetApp’s FlexClone feature Simpana MediaAgent copies the database data from the NetApp system to secondary
storage. It is possible to configure Simpana in a way that only selected snapshots are copied to secondary storage.
The snapshots (and secondary storage) will be aged automatically using Simpana’s Data Aging feature. The
recommendation is to assign a short retention time to the snapshot backups (e.g. 1-3 days) and keep the copied
snapshots for 2-4 weeks in order to serve restore requests targeting older database versions. The database logs
are saved to secondary media in a regular basis by Simpana without using IntelliSnap in the Server LAN. The
concept consists of scheduled backups of the database using the database modules in combination with
IntelliSnap. NetApp snapshots are performed several times a day. During the night a so-called Selective Full is
performed to tape.
In order to run restores, Simpana can copy the data back from the mounted snapshot or initiate a hardware
revert of production volumes to roll back to a specific snapshot. Simpana always performs SAP Restore and
Recovery (resp. recovery using other database agents). It can be done from secondary media, from snapshot by
copying to the original location or using the so-called SnapRestore feature.
All information necessary to use that concept is given in a Best Practice Paper “FlexFrame Backup with Simpana”
located on the FlexFrame Extranet – you may ask your Fujitsu Customer Support for getting a copy.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 34 of 37
ts.fujitsu.com/FlexFrame
15 Third-Party Software
The interfaces FlexFrame provides are basically the standard interfaces of the operating systems supported with
FlexFrame. The Operating Systems are installed in a way that they behave like standard OS installations for SAP.
One big difference is that the root file system "/" is mounted and accessed via NFS. The third-party software has
to be capable to deal with NFS-only file systems.
Some third-party software (e.g. system management tools, tools, etc.) may not be able to handle moving
(virtual) services and assume a fixed server-to-service relationship. In addition, those products may write files at
locations that are mounted read-only and must therefore be relocated to a read/write location.
During implementation of a FlexFrame project the required third-party products must be identified and a project-
specific analysis of this product must be initiated. Fujitsu offers such validation of third-party software as a
service.
As mentioned it is also possible with the FlexFrame Agents to integrate “third-party services” into the autonomy
scenarios. For this purpose a generic service is defined through a set of parameters which are used for its
identification and which generate the service states.
16 Support
16.1 Fujitsu Customer Support
FlexFrame environments are fully integrated into the support structure of Fujitsu. A large number of consultants
are available for design, planning, installation and further customer support.
16.2 SAP
The FlexFrame infrastructure solution approach is well-known and fully supported by SAP. Customers can create
problem reports through existing SAP mechanisms (SAP Support Portal) in the same way as for conventional SAP
system environments. Fujitsu are fully integrated and are informed by SAP if a problem is platform-specific.
Support staff from Fujitsu work on such problems using SAP’s support structures.
During installation of a SAP environment (FlexFrame or not) it is advisable to open TELNET access and a SAP
Support Portal connection to the installed system. If a problem occurs, the customer can open the SAP Support
Portal connection (with TELNET or SAPGUI) and provide logon information for support purposes. This remote
support allows rapid detection and correction of problems.
17 Benefits of the FlexFrame® Platform
The following sections briefly describe some of the benefits of the FlexFrame general concept.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 35 of 37
ts.fujitsu.com/FlexFrame
17.1 Shared Use of Storage Space
Legacy installations require dedicated LUNs for data volumes. Since growth of the data volumes is hard to predict,
a lot of free space is required to assure continuous operation of the database.
With the NAS systems as storage, multiple databases (SIDs) can share the same space (volume), and the disk
space is utilized much more efficiently.
17.2 Shared Usage of Servers
Due to the virtualization of SAP services, the total number of servers required can be reduced. Following are two
sample scenarios.
17.2.1 Less Servers Required for High Availability
Legacy concepts usually require a pair of dedicated servers for clustering SAP databases or central instances. In a
larger SAP environment, multiple cluster pairs of this type lead to a large number of servers just to cover
hardware failures. Usually the capacity installed is much larger than the capacity used.
In FlexFrame, spare servers are used to take over failed services from the complete FlexFrame environment, or, if
server pools and groups are used, from the complete pool/group, or in case of pool independent spare servers
even from the complete FlexFrame server park.
17.2.2 Better Coverage of Performance Peaks
In larger SAP environments, multiple SAP systems (SIDs) communicate with each other. This may result in the
situation where an ERP system processes, for example, month-end calculations and passes on information to a
BW system. This BW system now builds aggregates of the new information. The servers for both the ERP and BW
systems must be sized to be able to handle the maximum load. In legacy system environments both servers may
be idle most of the time.
With FlexFrame, the available hardware can be shared between processes. Processes can be moved to a faster
host when the load increases, and moved back to slower hardware after a load peak has been processed. The
number of high-end servers that are able to handle maximum peaks can therefore be significantly reduced.
17.3 Less Administration
The shared operating system feature provides advantages in an administrator’s daily life.
Linux Application Nodes share their "/" root file system. Hence, updates of software packages are available
immediately to all Application Nodes. This means that, after updating and testing with a Test Application Node,
you can activate a new root file system version by simply rebooting the Application Nodes.
17.4 Less Downtime during a Change of OS
Imagine you want to update a large number of servers to a new version of Linux. Usually this means creating
backups of each server, and updating and testing each server individually.
With FlexFrame you can simply create a new directory on the NAS storage and install your new version of the OS
in this directory.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 36 of 37
ts.fujitsu.com/FlexFrame
Once tested sufficiently, you only change the boot configuration to use the new root file system, and you reboot
the Application Nodes whenever convenient. And there is a most simple way of fallback to the old OS version if
needed.
17.5 Improved Change Management
An improved change management in FlexFrame environments is guaranteed by
dynamic SAP operation - you have a very high level of adaptivity and short response times in fulfilling
requirements
the higher quality of service - possible errors in making changes are minimized and downtimes are
reduced
low-cost adaptations - personnel expenses and costs for external service providers and capital spending
are reduced
Simple further development of the SAP environment - the standardization of the IT infrastructure
minimizes the time and work involved in adaptations, FlexFrame scales very easily and virtually without
limits and systems and new services can be rolled out at little cost and effort.
17.6 Simplified Multi-Client Operation
If services are provided for multiple customers, a conventional approach requires a separate infrastructure for
each one. As a result, the topics and problems in planning, implementation, operation and adaptation multiply.
Now FlexFrame is the ideal platform for supporting multiple clients with a single IT infrastructure or running
different SAP systems. The various environments are kept strictly apart from each other by separate networks. It
is also possible to reassign hardware between different customers or systems in a matter of minutes. Numerous
functionalities make it possible to configure SLAs for the individual customers to handle individual agreements.
Functions such as event and error management, accounting management, performance and capacity
management and convenient reporting options additionally support operations management for various
customers.
17.7 No Standby Database Server Required
A shadow database is commonly used for an additional level of high availability. An additional server is required
for such a concept. The changes to the database ("redo logs") are shipped to this remote server and are applied
to a copy of the database with a defined time lag. That means the shadow database is always some hours behind
its master DB. If a logical error occurs in the original database, the shipping of the logs is stopped and the
"waiting" logs will not be applied. If the error occurred before the fixed time delay (e.g. three hours), a restore
from other media is required. Most decision-making processes take a long time to decide whether to switch over
to the standby database or fix the issue in a different way.
Since FlexFrame is based on NAS storage systems that enable large amount of snapshots per volume to be
created, a more granular recovery procedure has been provided. It is possible to separate logs and data files by
volume. You can make rotating snapshots, e.g. on an hourly basis. If an error occurs the database will be stopped.
A snapshot from a time before the error happened is restored only for the data volume (or even on data file
level). Since the log files are still available, it is now possible to recover to a time just before the error occurred.
WHITE PAPER FLEXFRAME ORCHESTRATOR, Version 1.4A, Issue February 2018
Page 37 of 37
ts.fujitsu.com/FlexFrame
In FlexFrame, the additional server for an e.g. Oracle or DB2 Standby Database is not necessary.
18 FlexFrame® and SAP® Business Suite
FlexFrame is a platform explicitly designed for the SAP® Business Suite. The components supported by the
FlexFrame infrastructure solution are limited by SAP’s product availability matrix (PAM) and the managed systems
supported by SAP’s LVM.
Components which are not supported by FlexFrame should run on separate, dedicated servers. FlexFrame Agents
do not monitor such components unless these are virtualized services integrated into the autonomy monitoring
and reaction scenarios by means of the generic services (see 7.2).
The databases supported by FlexFrame are MaxDB (SAP DB), Oracle, DB2, SYBASE ASE, SYBASE IQ and HANA.
The SAP services currently supported by FlexFrame are TREX, MDM, Business Objects, Business Objects
Enterprise, SMD, Content Manager, WebDispatcher and HANA.
CONTACT
FUJITSU
All rights reserved, including intellectual property rights. Technical data subject to modifications and delivery subject to
availability. Any liability that the data and illustrations are complete, actual or correct is excluded.
Designations may be trademarks and/or copyrights of the respective manufacturer, the use of which by third parties for their
own purposes may infringe the rights of such owner.
For further information see ts.fujitsu.com/terms_of_use.html
Copyright © Fujitsu Limited 2018

Navigation menu