Server Operations Guide

server_operations_guide

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 125 [warning: Documents this large are best viewed by clicking the View PDF Link!]

Server Operations Guide
Table of Contents
Chapter 1. Introduction
Chapter 2. Server Configuration with bw.xml
Chapter 3. Admin Tools
Chapter 4. Watcher (Web Interface)
Chapter 5. Fault Tolerance
Chapter 6. Backups and Disaster Recovery
Chapter 7. Controlled Startup and Shutdown
Chapter 8. Stress Testing with Bots
Chapter 9. Security
Chapter 10. BigWorld Server Across Multiple Machines
Chapter 11. Multiple BigWorld Servers in a Single LAN
Chapter 12. DBMgr MySQL Support
Chapter 13. RPM
Chapter 14. First Aid After a Crash
- 14.1. Procedure For Crash Reporting
- 14.2. Q & A
Chapter 15. Common Log Messages
- 15.1. Warnings
- 15.2. Errors
Chapter 16. Clock
- 16.1. BigWorld Timing Methods
- 16.2. Linux Clock Source

BigWorld Technology 1.9.1. Released 2008.

Software designed and built in Australia by BigWorld.

Level 3, 431 Glebe Point Road

Glebe NSW 2037, Australia

www.bigworldtech.com

This document is proprietary commercial in confidence and access is restricted to authorised users. This document is

protected by copyright laws of Australia, other countries and international treaties. Unauthorised use, reproduction or

distribution of this document, or any portion of this document, may result in the imposition of civil and criminal penalties as

provided by law.

Table of Contents

1. Introduction .................................................................................................................................. 5

2. Server Configuration with bw.xml ................................................................................................ 6

2.1. The entry parentFile ....................................................................................................... 6

2.2. User dependent configuration ............................................................................................. 7

2.3. Command-Line Options ...................................................................................................... 7

2.4. General Configuration Options ........................................................................................... 7

2.5. Network Configuration Options ........................................................................................ 13

2.6. Load Balancing Configuration Options .............................................................................. 14

2.6.1. Entities Cardinality Configuration Options ............................................................. 14

2.7. BaseApp Configuration Options ........................................................................................ 15

2.7.1. Secondary Database Configuration Options ............................................................ 20

2.7.2. Packet Log Configuration Options .......................................................................... 20

2.7.3. ID Configuration Options ....................................................................................... 21

2.7.4. Client Upstream Limits ........................................................................................... 21

2.8. BaseAppMgr Configuration Options ................................................................................. 22

2.9. Bots Configuration Options ............................................................................................... 23

2.10. CellApp Configuration Options ....................................................................................... 25

2.10.1. Noise Configuration Options ................................................................................ 30

2.10.2. ID Configuration Options ..................................................................................... 30

2.10.3. CellApp Profiles Configuration Options ................................................................ 31

2.11. CellAppMgr Configuration Options ................................................................................ 32

2.12. DBMgr Configuration Options ........................................................................................ 35

2.12.1. Data Consolidation Options .................................................................................. 39

2.13. LoginApp Configuration Options .................................................................................... 39

2.14. Reviver Configuration Options ........................................................................................ 42

2.14.1. Reviver's BaseAppMgr Configuration Options ...................................................... 43

2.14.2. Reviver's CellAppMgr Configuration Options ....................................................... 43

2.14.3. Reviver's DBMgr Configuration Options ............................................................... 43

2.14.4. Reviver's LoginApp Configuration Options ........................................................... 44

3. Admin Tools ................................................................................................................................ 45

3.1. WebConsole ...................................................................................................................... 45

3.1.1. Modules ................................................................................................................. 45

3.1.2. Configuration ......................................................................................................... 49

3.1.3. How to Start ........................................................................................................... 49

3.2. Logger Daemons ............................................................................................................... 50

3.2.1. MessageLogger ....................................................................................................... 50

3.2.2. StatLogger .............................................................................................................. 55

3.3. Server Command-Line Utilities ......................................................................................... 67

3.3.1. control_cluster.py ........................................................................................... 67

3.3.2. eload (Entity Loader) ............................................................................................. 69

3.3.3. MessageLogger Related Utilities ............................................................................. 69

3.3.4. mls (Machine List) ................................................................................................. 70

3.3.5. runscript ............................................................................................................. 70

3.4. Standard GUI Applications ............................................................................................... 70

3.4.1. BWPanel ................................................................................................................. 71

3.4.2. Space Viewer .......................................................................................................... 75

4. Watcher (Web Interface) ............................................................................................................... 84

5. Fault Tolerance ............................................................................................................................ 86

5.1. CellApp Fault Tolerance .................................................................................................... 86

5.2. BaseApp Fault Tolerance ................................................................................................... 86

5.2.1. Distributed BaseApp Backup Method ..................................................................... 87

5.2.2. Non-Distributed BaseApp Backup Method .............................................................. 87

5.2.3. IP Address Change ................................................................................................. 89

5.2.4. Further Considerations ........................................................................................... 89

iii

5.3. Fault Tolerance with Reviver ............................................................................................. 90

5.3.1. Specifying Components to Support ......................................................................... 90

5.3.2. Command-Line Options .......................................................................................... 91

6. Backups and Disaster Recovery ................................................................................................... 92

6.1. Disaster Recovery .............................................................................................................. 92

6.2. Database Snapshot Tool .................................................................................................... 92

6.2.1. Configuration ......................................................................................................... 93

6.3. Data Consolidation Tool .................................................................................................... 95

6.3.1. Skipping Data Consolidation .................................................................................. 95

6.3.2. Ignoring SQLite Errors ............................................................................................ 96

7. Controlled Startup and Shutdown ............................................................................................... 97

8. Stress Testing with Bots ............................................................................................................... 98

8.1. The Login Process ............................................................................................................. 98

8.2. Watcher Interface .............................................................................................................. 99

8.3. Python Interface .............................................................................................................. 100

8.3.1. Python Controller (bot_op.py) ............................................................................ 100

8.3.2. Methods and Attributes ........................................................................................ 101

8.4. Controlling Movement .................................................................................................... 104

8.4.1. NodeProperties Section ......................................................................................... 105

8.5. Extending Bots ................................................................................................................ 106

8.5.1. Creating New Movement Controllers .................................................................... 106

9. Security ...................................................................................................................................... 108

9.1. Security of the Server ...................................................................................................... 108

9.2. Server Ports ..................................................................................................................... 109

9.3. Blocking Ports and Related Security Considerations ......................................................... 110

10. BigWorld Server Across Multiple Machines .............................................................................. 111

10.1. How To Start ................................................................................................................. 111

10.1.1. WebConsole ........................................................................................................ 111

10.1.2. Auto Configuration Via control_cluster.py ................................................... 111

10.1.3. Hard-Coded Scripts ............................................................................................. 111

10.1.4. Manual Start ....................................................................................................... 111

10.2. How To Stop ................................................................................................................. 111

10.3. How To Monitor ............................................................................................................ 111

10.4. LoginApp and Scalability .............................................................................................. 111

11. Multiple BigWorld Servers in a Single LAN ............................................................................. 113

11.1. Keeping Processes Separate ............................................................................................ 113

11.2. Centralised Cluster Monitoring ...................................................................................... 113

11.3. Auto-Detection of LoginApps ........................................................................................ 113

12. DBMgr MySQL Support ........................................................................................................... 114

12.1. Compiling DBMgr with MySQL Support ....................................................................... 114

12.2. Update bw.xml To Use MySQL ..................................................................................... 114

12.3. Initialise Database With Entity Definitions .................................................................... 114

12.4. Disabling Schema-Modifying Capability ........................................................................ 115

12.5. Enabling Secondary Databases ....................................................................................... 115

13. RPM ......................................................................................................................................... 116

13.1. Directory Structures and Files ........................................................................................ 116

13.2. How to Generate Binary RPM Packages ......................................................................... 116

13.3. Customising RPM Packages ........................................................................................... 117

13.4. Setting up a Yum Repository .......................................................................................... 117

13.5. Install, Upgrade and Uninstall using Yum Command ..................................................... 119

13.5.1. Install and Upgrade using a RPM Package Directly .............................................. 119

13.5.2. Install and Upgrade using Yum Repository .......................................................... 119

13.5.3. Remove an Installed Package ............................................................................... 119

13.6. How to Obtain Version Number of an Installed Package ................................................. 119

14. First Aid After a Crash ............................................................................................................. 121

14.1. Procedure For Crash Reporting ...................................................................................... 121

14.2. Q & A ........................................................................................................................... 121

15. Common Log Messages ............................................................................................................ 122

15.1. Warnings ....................................................................................................................... 122

15.2. Errors ............................................................................................................................. 123

16. Clock ........................................................................................................................................ 125

16.1. BigWorld Timing Methods ............................................................................................. 125

16.2. Linux Clock Source ........................................................................................................ 125

Chapter 1. Introduction

BigWorld Technology is BigWorld's middleware for implementing Massively Multiplayer Online Games.

This document is a guide to performing operations with the server software. It is not intended for game

designers or game logic implementers, but rather for 'machine room' or 'cluster control' operators and

administrators.

It is assumed that the server has been installed according to the instructions in the document Server

Installation Guide. An understanding of the basic BigWorld processes is also assumed. For more details on

these processes, see the document Server Overview's chapters Design Introduction and Server Components.

Note

For details on BigWorld terminology, see the document Glossary of Terms.

Chapter 2. Server Configuration with bw.xml

The single most important configuration file on the server is <res>/server/bw.xml, where <res> is the

resource tree used by the server (usually specified in ~/.bwmachined.conf).

All sever processes read this file. It contains many parameters, all of which are described in this chapter. The

default values are appropriate for many different games, and care should be taken when changing them,

since it might affect performance.

On the description of the parameters, please note the following:

▪Boolean parameters should be specified as true or false.

▪Where a tag is specified as tag1/tag2, the second tag is specified inside the scope of the first one. For

example, the tag dbMgr/allowEmptyDigest is specified as:

<dbMgr>

</dbMgr>

2.1. The entry parentFile

The entry parentFile in the configuration file specifies the next file in the chain of files where BigWorld

should look for an entry for a configuration option.

To assign a default value to a configuration option, BigWorld follows the steps below:

▪Searches for an entry for the configuration option in file <res>/server/bw.xml.

▪If the file does not contain an entry for the configuration option, then the chain of configuration files is

inspected, until the entry is found.

▪If the entry is not specified in any of these files, a hard-coded default is used.

All default values are stored in file bigworld/res/server/defaults.xml. Typically, the entry

parentFile in file <res>/server/bw.xml is set to server/defaults.xml, and only non-default

options are stored in your server/bw.xml.

The example below shows the configuration option maxCPUOffload in section balance having its default

value overridden:

User dependent configuration

Overriding default values for configuration options

2.2. User dependent configuration

A file that is user dependent can be used instead of bw.xml. This is useful to allow multiple users to run

from the same resource tree. In a production environment, for example, you may run the resources using a

test user before using the production user.

If a file with the name server/bw_<username>.xml exists, this is used as the start of the server

configuration chain instead of server/bw.xml.

Typically, the parentFile section in this file would refer to server/bw.xml and only options specific to

the user, such as dbMgr/databaseName would be in this file.

2.3. Command-Line Options

The configuration options specified in file <res>/server/bw.xml can also be overridden via command-

line arguments.

To override a default value, add arguments in the format +optionName value.

The example shows the baseApp section's configuration option pythonPort having its default value

changed to 40001, and the option archivePeriod changed to 0:

baseapp +baseApp/pythonPort 40001 +baseApp/archivePeriod 0

Values changed via the command line are not sent to components started via BWMachined. This includes

using WebConsole, control_cluster.py, and components started by a Reviver process.

2.4. General Configuration Options

The list below describes the general configuration options:

▪bitsPerSecondToClient (Integer)

Desired default bandwidth from server to the client. To calculate the number of bytes to be sent in each

packet, the formula below is used (where UDP_OVERHEAD is 28 bytes):

packetSize = (bitsPerSecondToClient / 8 / gameUpdateHertz) - UDP_OVERHEAD

▪debugConfigOptions (Integer)

Level of logging information generated when processing configuration parameters in file <res>/

server/ bw.xml.

The possible values are described in the list below:

▪0

No log is generated.

▪1

A log message is generated for each configuration option read.

▪2

General Configuration Options

A verbose message is generated for each configuration option read.

▪desiredBaseApps (Integer)

Number of BaseApps that need to be running for the server to start.

▪desiredCellApps (Integer)

Number of CellApps that need to be running for the server to start.

▪externalInterface (String)

Network adapter interface to use for external communication, if not explicitly set by the server component.

In a production environment, BaseApps are recommended to have two Ethernet adapters: one adapter

connected to the Internet, and a separate one connected to the internal LAN.

During development, there is no problem with using the same interface.

Accepted formats are:

▪Adapter name ‐ Examples: eth0, eth1

▪IP[/netmask] ‐ Examples: 10.5.2.1, 10.0.0.0/8, 192.168.5.0/24

▪Domain name ‐ Examples: intern0.cluster/24, extern5.cluster/24

This value can be overridden by a tag with same name in the following sections:

▪baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .

▪loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .

▪externalLatencyMax (Float)

Maximum number of seconds by which packets sent from the server process will be artificially delayed.

Each packet will be randomly delayed between this value and externalLatencyMin.

This value can be overridden by a tag with the same name in the following sections:

▪baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .

▪loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .

This feature is useful for testing during development.

See also the Stop the World button.

▪Stop the World

Stops all components of the running BigWorld server started by the following buttons:

▪Start single-machine world

▪Add CellApp

▪Add BaseApp

It does not stop BWMachined, which should be running all the time, nor tools such as the watcher process

or GUI apps (Space Viewer, General Meter, etc...).

The terminals are left open so that final output from the components can be examined.

The Close Terms button closes the terminal windows associated with the world.

3.4.1.2. Adding Buttons to BWPanel

When launched, BWPanel reads all files with extension .but in its folder. These files are shell scripts, each

defining a single button.

BWPanel

The code fragment below illustrates the implementation of the Display Details button by the

View_Info.but file:

#!/bin/sh

#BWPANEL 2.5 'Display Details'

# start the cell up in a terminal.

$MF_ROOT/bigworld/tools/myscript.sh

Example View_Info.but file — Implementation of the Display Details button

As displayed in the example code above, the button definition file has the sections below:

▪Code interpreter

Program to use to run the script

▪Display information

Optional display information. The parameters are described in the list below:

▪Button order

Floating-point specifying the order in which button is to be displayed.

Button will not be displayed if value specified is not unique.

▪Button label

Label to be displayed for the button.

If you choose to not specify it, then it will be derived from the file name (underscores are converted to

spaces, and the extension is dropped).

In the example above, if Display Details was not specified, then the button would have been labelled

View Info (the file name is View_Info.but)

▪Code

Code to be executed when button is clicked — the syntax is specific to whatever interpreter was chosen.

3.4.1.3. Command-Line Options

Located under the bigworld/tools/server/bwpanel folder, BWPanel can be called via command line

with the following syntax:

bwpanel [-display display]

The options for bwpanel are interpreted by Tcl/Tk's wish command, and are described in the list below:

▪-display display

Specifies the display (and screen) on which to display window.

The display argument specifies the X window on which to display instead of the one specified in the

DISPLAY environment variable.

Space Viewer

3.4.2. Space Viewer

Space Viewer (located in bigworld/tools/server/space_viewer) logs and displays a dynamic graphic

representation of the distribution of cells on a space, and the entities (player characters, NPCs, and other

objects) on those cells.

Space Viewer uses a client/server architecture, and is therefore composed of two distinct parts, which may

be used simultaneously or separately. The "server" side either communicates with the BigWorld server to

collect and log information about a space, or reads a previously written log. The "client" side connects to the

server process and displays the space information it provides.

Since the two halves of Space Viewer communicate via TCP, they can be run on different machines, although

obviously the client is most responsive when both parts are run on the same machine.

The most common use of Space Viewer is generally to monitor the state of a live BigWorld server, and as such,

both the client and server components are run in tandem. Other times, when space data needs to be collected

over long periods of time (for example, during testing), the server part of Space Viewer (svlogger.py) will be

run on its own, with the option to connect a client window to it at any time to examine the current state or

look back through the log recorded thus far.

In the most common case of merely wanting to monitor an active server, and not being interested in

extended period logging, Space Viewer will be started by simply running space_viewer.py. Its main window

is displayed below:

Space Viewer window

3.4.2.1. Selecting Spaces to View

SpaceViewer displays a list of user IDs that are currently running BigWorld server clusters. If you do not see

your UID listed in Space Viewer, then select File → Refresh menu item — this is necessary because the list

is not automatically refreshed.

In the example above, users 502, 573 and 589 are running server clusters.

Clicking the icon to the left of user name expands the list to display all spaces available for that server instance.

In the example above, the server cluster run by user 502 has spaces 2, 5 and 9. To see the map of a space, double

click its entry on the list. This will start an svlogger process logging data from the selected space, as well as a

client window that is connected to it. Closing the client window will cause the logger to terminate and exit.

3.4.2.2. Menu Items

The list below describes the menu item available in the Space window:

Space Viewer

▪File → Close window and terminate Logger

Closes the Space window and terminates the Logger. This item does not quit the Space Viewer.

▪File → Close window

Closes the Space window. This item does not quit the Space Viewer.

▪View → Update entities

Specifies that entities' position is determined by polling the currently selected cell.

Note: It is possible to view entities from only one cell at a time.

▪View → Set Cell App Update Frequency

Opens the Change Display Update Interval dialog box, where you can change how often entity positions

are queried from the cell.

Smaller values for this settings cause greater load on the cell. In large systems, it is recommended to

increase the delay (e.g., to 3 seconds)

▪View → Set Cell App Manager Update Frequency

Opens the Change Display Update Interval dialog box, where you can change how often the cell

boundaries are queried from the CellAppMgr and redrawn.

▪View → Zoom in

Displays a smaller area of the space.

▪View → Zoom out

Displays a bigger area of the space.

▪View → Zoom to space bounds

Automatically zooms to the extents of the current space.

▪View → Image overlay → Display image overlay

Opens the Choose An Image To Overlay dialog box, where you can choose the file with the image to

superimpose onto the display.

The image would typically be a screenshot from inside WorldEditor.

Examples are provided in fantasydemo/res/server/space_images.

▪View → Image overlay → Remove image overlay

Disables the display of an image overlay.

▪View → Image overlay → Recent image overlays

List of most recent image overlays.

▪View → Graph overlay → Display graph overlay

Opens the Choose A Graph File dialog box, where you can choose the file with the graph topology to

superimpose onto the display.

Space Viewer

File should have the XML format accepted by the standard bots Patrol movement controller. For more

details, see “Controlling Movement” on page 104 .

The user will be prompted to specify the factor by which the specified graph should be resized.

This feature is useful when you are running tests with bots.

Sample graph topologies can be found in fantasydemo/res/server/ bots.

▪View → Graph overlay → Remove graph overlay

Disables the display of a graph overlay.

▪View → Graph overlay → Recent graph overlays

List of most recent graph overlays.

▪Colour → CellApp ID, Colour → IP address, Colour → Cell load, Colour → Partition load, Colour

→ Entity bounds, Colour → Space boundary, Colour → Grid, Colour → Ghost entity, Colour → Cell

boundary

Lets you select a colour to draw the specified element in.

A sub-menu displays the colours available for rendering the element.

If you do not want the item to be displayed, then select None.

▪Colour → Relative colouring

Colours the CellApp load information according to its load.

Colour will range from aqua blue (light loaded) to full red (heavy loaded).

Light-load CellApp and Heavy-load CellApp

▪Entity size → 25%, Entity size → 50%, Entity size → 75%, Entity size → 100%, Entity size → 150%, Entity

size → Custom scale...

Sets the size of the entity symbol to the specified scale.

If the Custom Scale menu item is selected, then a dialog will open, letting you specify a value ranging

from 1 through 1,000.

▪Util → Retire selected cell

Item available mainly for debugging purposes.

Removes the selected cell from the system.

Note that CellAppMgr will probably add the cell back when it performs auto load balancing.

▪Util → Stop selected CellApp

Item available mainly for debugging purposes.

Space Viewer

Stops the selected CellApp process.

▪Visible entities → Show all

This will set all entity types in the selected cell as visible, giving a complete view of all entities.

▪Warp → Warp to time

Opens the Warp dialog box, where you can specify the date and time for which you want to display space

data.

▪Warp → Warp to log event

Opens the Choose A Timestamped Log dialog box, where you can specify the log file containing the space

data that you want to display.

▪Help → Help

Displays help for Space window.

3.4.2.3. Viewing Spaces

Multiple spaces can be viewed simultaneously, each in their own window. To view a space, double-click its

entry on the space list displayed in Space Viewer.

The example below shows a space being handled by five cells. It is important to note that in the selected cell

only the entity types selected in the Visible Entities menu item will be displayed. All ghost entities will be

drawn regardless of the entities selected in this menu.

Initially, no cell on the space will be selected for viewing. The cell boundaries and the cell information (IP

address, load, etc...) will always be shown, but to see the entities on a particular cell (as with cell #1 in the

screenshot below) it must be selected with a single click

Space Viewer

Space window

The list below describe some of the components of the Space window:

▪CellApp load

CPU load calculated by the CellApp itself.

▪Selected cell's real entity

The dots are coloured according to their type. For more details see “Customising Entity and Display

Colours” on page 80 .

▪Scale information

Indicates the size of the viewed area (space and cells). In the example above, the space is slightly over

3km x 4km.

▪Cell boundary

If the displayed space contains more than one cell, then the geographic tessellation will be indicated.

▪Cursor position

Displays the current position of the cursor (x,z) in metres.

▪Entities information

Displays the number of real and ghost entities on selected cell.

To change the visualisation of the space via keyboard or mouse, you have the following options:

▪Select a cell

Click anywhere in a cell to make it active. Click on the cell again to deselect it.

▪Move the view

Click within the display and drag the mouse to move the cell display.

▪Zoom in/out

To zoom in and out of the display, use the Page Up and Page Down keys, or the mouse wheel.

You can also use the middle mouse button to draw a rectangle, thus selecting the new area to view. As

you zoom, scale information is updated to reflect the changes. If you get lost, restart the visualisation by

selecting the View�Zoom To Space Bounds menu item, or by pressing Ctrl+Home.

You can also seek the space view forwards or backwards in time using the keyboard. You can seek backwards

as far as the time the logger was started, and if you seek forward to the current time, then the window will

start updating in real time again.

The list below displays the keyboard shortcuts for controlling playback:

▪LEFT ARROW

Moves the playback 1 second backwards.

▪RIGHT ARROW

Moves the playback 1 second forward.

Space Viewer

▪UP ARROW

Moves the playback 60 seconds backwards.

▪DOWN ARROW

Moves the playback 60 seconds forward.

▪Home

Moves the playback to the beginning of the log.

▪END

Moves the playback to the end of the log.

▪Ctrl+L

Prompts the user to select a command log file — this can be any file in which each line contains date/time

string like 'Thu Nov 17 17:45:45 2005'..

Once the file is selected, the script will display all logged events contained in the log file, and prompt the

user to select one.

Once event is selected, playback time will be warped to the time of selected event.

▪Ctrl+W

Prompts the user to enter a time to warp the playback to.

In order to facilitate identification of entities there is also tool tip information for all non-ghosted entities.

This information will appear if the cursor hovers over an entity marker for a sufficiently long time.

Entity tooltip

The entity type name is retrieved from file <res>/scripts/entities.xml, and corresponds to the IDs

in the menu Visible Entities described above.

3.4.2.4. Customising Entity and Display Colours

Space Viewer allows you to customise the colours to be used when drawing the elements in the Space

window. But this customisation is not saved, and will not be in effect the next time the application is started.

However, it is possible to permanently change the colours by customising a Python file.

Entities in the current cell are displayed with a coloured dot. The colour can be customised in the local file

style.py. The simplest way is to update the dictionary colours (which contains RGB colour definitions for

each entity type).

The code fragment below illustrates the definition of permanent entity colours:

# specify colour for entity Types.

Space Viewer

entityColours = {

0: ((111, 114, 247), 2.0),

1: ((244, 111, 247), 1.0),

2: ((124, 247, 111), 1.0),

3: ((114, 241, 244), 1.0),

4: ((21, 168, 179), 1.0)

}

Example local file style.py

The entity's number is defined by its order of appearance in file <res>/scripts/entity.xml. Entities

which ID is not defined in the dictionary are coloured with RGB (255,192,192).

So assuming the following file <res>/scripts/entities.xml, these will be the entities' display colours:

<root>

</root>

Example file <res>/scripts/entities.xml and their display colours in Space Viewer

You can also replace the entire function getColourForEntity(entityID, entityTypeID), using your own

algorithm to colour the entities.

All other colours are defined by the dictionary colourOptions. They can be changed to any of the colours

defined by the colours array.

3.4.2.5. Running the Client and Server Parts of Space Viewer Separately

As described at the start of this section, Space Viewer's two halves can be run separately as needed. This

would usually take the form of running svlogger over an extended period of time, connecting clients to it to

monitor live state as necessary, then eventually shutting down the logger. Later, the logger can be connected

to its previously recorded log (instead of a live server) and client windows can be attached to it to replay

the log data.

3.4.2.5.1. svlogger.py

The svlogger script provides log writing and reading for Space Viewer. In writing mode, it is responsible for

gathering the space data from the actual running BigWorld server (which it does by polling the CellAppMgr

every second or so). In reading mode, it opens a previously recorded log for replay via any client connections.

The basic options for svlogger include:

▪-l <listenip:listenport>

Specifies the port that svlogger will listen for connections from svreplay. If not specified, any available

port will be used.

Space Viewer

▪-o <logdir>

Specifies the directory to write the data log to. Default is /tmp/svlog.

▪-p <prefix>

Specifies the prefix for the log files. Default is svlog.

▪-s <spaceid>

Specifies the space to log data for. Default is the server’s default space.

▪-u <uid>

Specifies the user to log data for. Default is current user's ID.

▪-k | --keepalive

Specifies that the logger should terminate once the last client connection is terminated. Useful when only

interested in monitoring a live server, and not interested in logs.

▪-r <logpath>

Activates replay mode for the log stored at the given path.

The path is the concatenation of the <logdir> and <prefix> specified for -o and -p switches,

respectively. For example, if you want to replay the default log location, you would specify /tmp/svlog/

svlog as the <logpath>.

This option can also be passed to space_viewer.py to open a log with svlogger and automatically

connect a viewer window to it.

3.4.2.5.2. Connecting a Client Window to svlogger

If you have started svlogger manually, then you will also need to manually connect client windows to the

logger if you wish to view its space data. This is easily done by running space_viewer.py with the -c

<ip:port> option. The address passed to this option should be the address that svlogger is listening for

connections on (which is displayed in the initial output from svlogger after it starts up).

3.4.2.6. Command-Line Options

Located under folder bigworld/tools/server/space_viewer, space_viewer.py can be called via

command line with the following syntax:

space_viewer.py [{-u |--uid=}<user_id>]

[{-s |--space=}<space_id>]

[{-c |--connect=}<ip:port>]

[{-r |--replay=}<log_prefix>]

[{-d |--dump-db=}<log_db>]

[--autoselect]

[-h|--help]

The options for space_viewer.py are interpreted by Tcl/Tk's wish command, and are described in the list

below:

▪{-u |--uid=} <user_id>

User for whose space is to be viewed. If this option is used, then -s|--space must also be specified.

▪{-s |--space=} <space_id>

Space Viewer

Space to view. If option -u|--uid is not specified, then the viewed space_id will be that of current user.

▪{-c |--connect=} <ip:port>

Address on which to connect to svlogger.py. For details, see “Connecting a Client Window to svlogger”

on page 82 . If this option is specified, the -u option must not be specified.

▪{-r |--replay=} <log_prefix>

Opens log with svlogger and automatically connects a viewer window to it.

▪{-d |--dump-db=} <log_db>

Displays all values from log_db.

▪--autoselect

On startup, automatically selects a cell on specified space. If this option is used, then -s|--space must

also be specified.

▪-h|--help

Displays online help.

Chapter 4. Watcher (Web Interface)

A Watcher is a general mechanism that exposes internal operational parameters of a running BigWorld

Server, so that an administrator can view and change them.

Note

As of BigWorld 1.8, this tool is replaced by the new tool WebConsole. For details, see

“WebConsole” on page 45 .

The Watcher web interface process collects watcher data and publishes it through an HTTP interface. It is

mainly used to diagnose and rectify an isolated problem once it has been discovered. For details on how to

instrument your own server processes and extensions, see Server Programming Guide, chapter The Watcher

Interface.

Located on bigworld/tools/server, the watcher process listens for UDP broadcast messages from the server

components containing announcements of new sources of watchers. It then publishes this information via

an HTTP interface. When links are clicked on, the interface performs deeper watcher requests on behalf of

the user.

To start a watcher process, execute the following command on any machine in the server cluster:

$MF_ROOT/bigworld/tools/server/watcher &

The watcher process listens for HTTP requests on port 20017. To view watcher information, just open a

browser (on Windows or Linux) on address http://yourhostname:20017. The hyperlinks give you access to

the watcher tree.

Note

Each watcher only shows the components registered under the UID that was used to

start the watcher process — client machines are shown on all watchers, since they run

on Windows boxes, and therefore do not have Unix UIDs.

You can run multiple watcher processes on the network.

The illustrations below display a watcher tree being traversed on a web browser:

Web page with CellAppMgr's loadSmoothingBias watcher

Chapter 5. Fault Tolerance

The Fault Tolerance system in BigWorld is designed to transparently cope with the loss of a machine due to

physical, electrical, or logical fault.

The game systems in general should not be affected, and the game experience will continue normally for

most clients, with possibly a brief interruption for clients closest to the faulty machine.

The loss of any single machine (running a single process) is always handled. The loss of multiple machines in

a short time frame may not always be adequately handled. In such extreme cases — for example, a complete

power outage affecting all machines — the Disaster Recovery system will be invoked. For more details, see

“Disaster Recovery” on page 92 .

Note that, in spite of handling failures, the Fault Tolerance system should not be relied upon to cover up a

software bug that causes a component to crash. All such bugs should be found and fixed in the source code.

If the bug is believed to be in BigWorld Technology code (and not in a customer's extension), then contact

BigWorld Support with details of the problem. For more details, see First Aid After a Crash on page 121 .

5.1. CellApp Fault Tolerance

CellApp fault tolerance works by backing up the cell entities to their base entities.

As long as a backup period is specified for the cell entities, the fault tolerance for the CellApp processes is

automatic. An operator should ensure that there is enough spare capacity in available CellApps to take up

the load of a lost process.

For details on how to specify the CellApp backup period, see “CellApp Configuration Options” on page 25 .

Note

Cell entities without a base entity are not backed up, and therefore will not be restored

if their process is lost.

Note

Since cell entities back up to base entities, running CellApp and BaseApp processes on

the same machine should be avoided. If the entire machine is lost, this level of fault

tolerance will not work.

For more details on the implementation of CellApp fault tolerance on code level, see the document Server

Programming Guide's chapter Fault Tolerance.

5.2. BaseApp Fault Tolerance

The BaseApp supports a choice of two schemes for fault tolerance:

▪Distributed BaseApp Backup

Each BaseApp backs up its entities on other (more than one) regular BaseApps.

▪Non-distributed BaseApp Backup

Regular BaseApps back up all their entities on dedicated Backup BaseApps.

Note

As of BigWorld 1.9, non-distributed BaseApp is deprecated. It is planned that support

for this feature will be removed in BigWorld 2.0.

Distributed BaseApp Backup Method

For more details, see the document Server Overview's section Server Components → BaseApp → Fault

tolerance. For details on the implementation of BaseApp fault tolerance on code level, see the document

Server Programming Guide's chapter Fault Tolerance.

The backup method is determined by bw.xml's configuration option baseAppMgr/useNewStyleBackup.

For details, see “BaseAppMgr Configuration Options” on page 22 .

Note

BigWorld recommends the Distributed BaseApp Backup method, for the following

reasons:

▪No need for dedicated Backup BaseApps.

▪Backup load is distributed over a period of time.

▪No need for IP switching, which might not be allowed by operating system and/or

router. This also makes cluster management easier.

5.2.1. Distributed BaseApp Backup Method

This fault tolerance method does not require differentiating between normal and backup BaseApps — all

BaseApps can perform both roles.

Each BaseApp is assigned a set of other BaseApps and a hash function from an entity's ID to one of these

backups. Over a period of time, all entities are backed up. This is then repeated. If a BaseApp dies, then the

entities are restored on to the appropriate backup.

Advantages of this method include that the backup can be done over a period of time, there is no need for

dedicated backup BaseApps or IP switching.

Disadvantages include the possibility of base entities that were previously on the same BaseApp end up on

different BaseApps (which places limits on scripting), and the fact that attached clients will be disconnected

on BaseApp failure, requiring a re-login.

Distributed BaseApp Backup — Dead BaseApp's entities can be restored to Backup BaseApp

5.2.2. Non-Distributed BaseApp Backup Method

In this method, a BaseApp can be started as either a normal BaseApp (which services requests and contains

bases), or as a Backup BaseApp. A Backup BaseApp stores bases and entities backed up by one or more

normal BaseApps, and takes over if one of them stops replying to watchdog pings.

The fact that Backup BaseApp can back up more than one BaseApp allows you to trade risk against hardware

cost when deciding how many Backup BaseApps to install.

Non-Distributed BaseApp Backup Method

To start a BaseApp as a Backup, use the command below:

baseapp -backup

If a BaseApp does not have a corresponding Backup BaseApp, then it will search for one on the network.

This is done as illustrated in the series of steps below:

1. Initial state

This example supposes a class C network x.x.x.0 as the connection to the Internet, and a private network

of 192.168.0.0, as illustrated below:

Initial state

2. BaseApp becomes unavailable

If BaseApp x.x.x.102 becomes unavailable, then all base mailboxes for bases on it will still refer to

the address 192.168.0.2, and Client5, Client6 and Client7 will have no proxy to communicate with, as

illustrated below:

BaseApp becomes unavailable

3. Backup BaseApp replaces unavailable BaseApp

Once the Backup BaseApp x.x.x.105 detects that it needs to replace the unavailable BaseApp, the following

steps will take place.

a. Backup BaseApp x.x.x.105 changes its internal and external IP addresses to the old BaseApp's ones.

b. From this point one, Backup BaseApp on x.x.x.105 becomes know on the network as BaseApp x.x.x.102.

IP Address Change

c. If the old machine is still responding, then new BaseApp x.x.x.102 issues a request to change its

addresses.

d. New BaseApp x.x.x.102 creates all entities stored from the original one, calling onRestore for each one.

e. BaseApp x.x.x.103 and the new BaseApp x.x.x.102 can find a new Backup BaseApp.

f. Any process communicating with the old BaseApp x.x.x.102 can keep doing so with the new one,

without any impact.

The final network configuration is illustrated below:

Backup BaseApp replaces unavailable BaseApp

5.2.3. IP Address Change

Since all network configurations have their own peculiarities, the process of changing IP addresses is

delegated to a shell script. However, the BaseApp first executes a program called change_ip, since changing

IP addresses is typically restricted to the root user, and Linux does not allow shell scripts to be launched

with suid root. This program calls setuid(0), and then executes the script ../scripts/change_ip.sh

relative to its working directory.

Because of this, you should ensure that appropriate permissions are set for each of these two files. The

permissions can be set with the commands below:

chown root bigworld/bin/scripts/change_ip.sh

chmod 700 bigworld/bin/scripts/change_ip.sh

chown root bigworld/bin/Hybrid/change_ip

chmod +s bigworld/bin/Hybrid/change_ip

The source code is located in folder src/server/tools/change_ip/main.c, and is compiled into

$MF_ROOT/bigworld/bin/$MF_CONFIG/change_ip.

The script change_ip.sh is expected to be customised for each particular installation. The script is called

with two command line parameters: the first is the interface to change the IP address for, and the second is

the new IP address for the machine.

5.2.4. Further Considerations

Things to consider while configuring BaseApp fault tolerance include:

▪Do your routers place any special requirements on the IP changeover, such as forcing ARP broadcasts, or

changing MAC addresses as well?

Fault Tolerance with Reviver

▪Do you need to have a separate monitoring network?

Both the internal and external BaseApp IP addresses are subject to change, and so it can be useful to have

a fixed IP address on the system, for administrative purposes. This may just be an alias on the internal

interface (established with the command call ipconfig eth1:0 192.168.0.1, for example.) This

could be very useful, for example, to ssh a machine with problems.

Due to a number of issues that can arise from changing the IP address of the machine, BigWorld has an

option to only pair backup BaseApps with BaseApps on the same machine, which removes the need to

change IP address of a machine. This is set by bw.xml's baseAppMgr/onlyUseBackupOnSameMachine

configuration option (for details, see “BaseAppMgr Configuration Options” on page 22 .).

5.3. Fault Tolerance with Reviver

Fault tolerance for BaseAppMgr, CellAppMgr, DBMgr, and LoginApp is provided by the Reviver, by starting

a new instance of the process to replace the unavailable one.

Although it is possible for one Reviver to watch all processes, it is recommended to run a few of them on

different machines, since Revivers normally stop after reviving a process.

For more details on Reviver, see the document Server Overview's chapters Design Introduction and Server

Components.

5.3.1. Specifying Components to Support

The file /etc/bwmachined.conf can contain entries for categories, and a list of items pertaining to it, with

the syntax below (the asterisk character —*— meaning 0 or more occurrences):

*[<category>]

*<item>

Please note that this setting can only be specified on the global configuration file.

Tools can query BWMachined to find the tags associated with categories. Categories may be used by future

tools or customer-created tools.

An example /etc/bwmachined.conf would have its bottom part as below (since the top part will contain

user settings, as described in the section above):

[Colours]

blue

green

red

When the Reviver starts, it queries the local BWMachined process, and will only support the components

that have an entry in a special <category> called Components on /etc/bwmachined.conf.

An example /etc/bwmachined.conf specifying that the Reviver should support all singleton server

components would have its bottom part as below:

[Components]

baseApp

baseAppMgr

cellApp

cellAppMgr

dbMgr

Command-Line Options

loginApp

Note

If the [Components] category does not contain any entries, then Reviver will support

all singleton server components.

BaseApp and CellApp will not be restarted by Reviver — the [Components] entries

are used by WebConsole and control_cluster.py to determine which processes

should be started by BWMachined on that host.

The configuration file is only read when BWMachined starts. It will have to be restarted if you want it to

acknowledge your changes.

5.3.2. Command-Line Options

The supported components can also be specified via command-line (even though we recommend that you

use /etc/bwmachined.conf for that), as below:

reviver [--add|--del {baseAppMgr|cellAppMgr|dbMgr|loginApp} ]

If reviver is invoked with no options, then it will try to monitor all singleton processes specified in category

Components of /etc/bwmachined.conf.

The options for invoking reviver are described in the list below:

▪--add { baseAppMgr | cellAppMgr | dbMgr | loginApp }

Starts the Reviver, trying to monitor only the components specified in this option. That means that the

components list in /etc/bwmachined.conf will be ignored.

▪--del { baseAppMgr | cellAppMgr | dbMgr | loginApp }

Starts the Reviver, trying to monitor all components specified in the list in /etc/bwmachined.conf,

except the ones specified in this option.

▪To start Reviver trying to monitor all processes specified in Components in /etc/bwmachined.conf:

reviver

▪To start Reviver trying to monitor only DBMgr and LoginApp:

reviver --add dbMgr --add loginApp

▪To start Reviver trying to monitor all processes specified in Components in /etc/bwmachined.conf,

except DBMgr and LoginApp:

reviver -del dbMgr -del loginApp

Chapter 6. Backups and Disaster Recovery

BigWorld's fault tolerance ensures that the server continues to operate if a single process is lost. The server

also provides a second level of fault tolerance known as disaster recovery that comes into play when multiple

server components fail at once and the server needs to be shut down.

For additional protection against exploits or bugs the database should be backed up regularly. The snapshot

tool facilitates making backups of the database.

6.1. Disaster Recovery

The server's state can be written periodically to the database. In case the entire server fails, then it can be

restarted using this information. To enable the periodic archiving of entities, an archive period needs to be

set via the configuration option archivePeriod for BaseApp and CellAppMgr processes. For more details,

see “BaseApp Configuration Options” on page 15 , and “CellAppMgr Configuration Options” on page 32 .

Disaster recovery only works when the underlying database is MySQL — it does not work when the XML

version is used. Please see DBMgr MySQL Support on page 114 for more information about about enabling

MySQL support in BigWorld.

Starting the server using recovery information from the database is the same as starting it from a controlled

shutdown. For more details, see Controlled Startup and Shutdown on page 97 .

For details on the scripting API related to disaster recovery, see the document Server Programming Guide's

chapter Disaster Recovery.

6.2. Database Snapshot Tool

For background information on the need for the snapshot tool see Server Programming Guide's chapter

Database Snapshot.

The snapshot tool is a Python script that facilitates making a backup copy of the database that combines

information from the primary database and secondary databases. It is located in bigworld/tools/

server/snapshot/snapshot.py. In its current form, it should be considered as example code. It should

be reviewed and customised for each environment.

The snapshot tool should be run on a separate machine to avoid interfering with the normal operation of

the server. The snapshot tool can use a significant amount of CPU and disk resources and does so in a fairly

spiky manner. We will refer to the machine running the snapshot tool as the snapshot machine.

The snapshot tool performs the following sequence of operations:

1. Runs the transfer_db command on each BaseApp. transfer_db performs a snapshot of the secondary

database and sends the snapshot to the snapshot machine using the rsync command.

2. Runs the transfer_db command on the primary database's MySQL server. transfer_db performs an LVM

snapshot of the primary database and sends the LVM snapshot to the snapshot machine using the rsync

command.

3. Starts a MySQL server on the snapshot machine and specifies its data directory to be the copy of the

primary database.

4. Runs the data consolidation tool on copies of the primary and secondary databases.

5. Archives the consolidated snapshot.

The snapshot tool logs all messages to message_logger. A snapshot should be considered invalid if there

is an error.

Configuration

The snapshot tool accepts a single command-line argument and command line options. The snapshot tools

help message is below:

Usage: snapshot.py [options] SNAPSHOT_DIR

Options:

-h, --help show this help message and exit

-b BWLIMIT_KBPS, --bwlimit-kbps=BWLIMIT_KBPS

file transfer bandwidth limit in kbps, default is

unlimited

-n, --no-consolidate skip consolidation, default is false

The list below provides some common examples of using the snapshot tool:

▪snapshot.py /home/bwtools/snapshots

Takes a snapshot. The snapshot is archived to/home/bwtools/snapshots.

▪snapshot.py /home/bwtools/snapshots --bwlimit-kbps=5000

Takes a snapshot. The snapshot is archived to/home/bwtools/snapshots. Each secondary and primary

database is transferred with a 5000 kbps bandwidth limit.

▪snapshot.py /home/bwtools/snapshots --no-consolidate

Takes a snapshot. The unconsolidated snapshot is archived to/home/bwtools/snapshots.

▪snapshot.py /home/bwtools/snapshots -n -b5000

Takes a snapshot. The unconsolidated snapshot is archived to/home/bwtools/snapshots. Each

secondary and primary database is transferred with a 5000 kbps bandwidth limit.

6.2.1. Configuration

For the snapshot tool to work, the following conditions must be met:

▪bwmachined must be running on the primary database machine. For background information see Server

Overview's chapter Server Components→BWMachined.

▪All the entity scripts and entity definition files must be present on the snapshot server.

.bwmachined.conf must be configured correctly for the user running the snapshot tool.

▪The snapshot section in /etc/bigworld.conf must be configured correctly on the primary database

machine.

▪The primary database is stored in an LVM volume and there is sufficient unpartitioned space on the

primary database machine to perform an LVM snapshot.

▪The directory the LVM volume is to be mounted to exists and has no other devices mounted to it.

▪The snapshot user must be able to connect to the snapshot machine using ssh from the primary database

machine and BaseApp machines without a password. For a user whose $HOME directory is on a NFS,

this can be done via the command-line as below:

$ ssh-keygen -t dsa

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Configuration

▪lvcreate, lvremove, mount, unmount and chmod must be installed and on the path of the primary

database machine.

▪mysqld_multi must be installed and on the path of the snapshot machine.

▪rsync must be installed and be on the path of the snapshot, the primary database and all the BaseApp

machines.

▪The snapshot_helper utility must be built. To build snapshot_helper, MySQL development files must

be installed. Please see “Compiling DBMgr with MySQL Support” on page 114 for details on how to

install MySQL development files. After installing MySQL development files, issue the make command

from within the bigworld/src/server/tools/snapshot_helper directory.

▪snapshot_helper has its setuid attribute set so user ID is root upon execution on the primary database

machine. This can be done via the command-line as below:

# chown root:root $MF_ROOT/bigworld/tools/server/snapshot/snapshot_helper

# chmod 4511 $MF_ROOT/bigworld/tools/server/snapshot/snapshot_helper

For security, all privileged commands (lvcreate, lvremove, mount, unmount and chmod) are invoked via

snapshot_helper which reads command arguments from the trusted /etc/bigworld.conf.

The keywords in the /etc/bigworld.conf are described in the list below:

▪datadir

The primary database MySQL server's data directory.

▪lvgroup

The name of the LVM volume group in which the primary database belongs to. The LVM snapshot volume

will also belong to this group.

▪lvorigin

The name of the origin LVM volume snapshotted. The MySQL data directory should be on this volume.

▪lvsnapshot

The name of the LVM snapshot volume that will be created and deleted by the snapshot tool. This name

is also use to generate the snapshot volume mount path, /mnt/lvsnapshot, which must already exists.

▪lvsizegb

The size of the LVM snapshot volume in gigabytes. The snapshot does not need the same amount of storage

the origin has as only the modified files are copied, in a typical scenario, 15-20% might be enough. If the

LVM snapshot volume becomes full it will be dropped ,so it is important to allocate enough space.

Below is an example /etc/bigworld.conf configuration file:

[snapshot]

datadir = /var/lib/mysql

lvgroup = VolGroup00

lvorigin = LogVol00

lvsnapshot = bwsnapshot

lvsizegb = 2

Note: The snapshot tool also reads <res>/server/bw.xml for the dbMgr/host, dbMgr/username,

dbMgr/password and dbMgr/databaseName.

Data Consolidation Tool

6.3. Data Consolidation Tool

The data consolidation tool is used to incorporate data from secondary databases into the primary database. It

is called consolidate_dbs and is located in bigworld/bin/Hybrid/commands. This tool is only provided

in source code form and must be built before use. Please see “Enabling Secondary Databases” on page 115

for details on how to build the data consolidation tool.

Once this tool is built, it will be automatically run during system shutdown or during system start-up

if the system was not shutdown correctly. There should be no need to manually run this tool except for

consolidating backups created by the snapshot tool (see “Database Snapshot Tool” on page 92 ). This

tool logs messages to Message Logger which can be viewed using the Log Viewer tool in Web Console.

This tool will accept two command-line arguments:

▪Primary Database

The parameters required to connect to the primary database should be specified in the form

host;username;password;database_name. When specifying this on the command-line, it is

necessary to quote the entire parameter to prevent the shell from treating it as multiple commands.

▪Secondary Databases

A file containing a newline separated list of fully qualified paths to secondary database files.

When run without command line arguments, it will use DBMgr's configuration options located

in <res>/server/bw.xml to connect to the primary database. It will use the data in the

bigworldSecondaryDatabase table to retrieve the secondary databases.

The data consolidation tool uses the transfer_db utility located in located in bigworld/bin/Hybrid/

command to transfer the secondary database files from the BaseApp machines to the machine where the data

consolidation tool is running. The transfer_db utility will be launched (via bwmachined) on each machine

that contains a secondary database file. The transfer_db utility then opens a TCP connection to the data

consolidation tool through which it will send the secondary database file.

The data consolidation tool will store the secondary databases locally in the directory specified by the

<dbMgr>/<consolidation>/<directory> configuration option. It will incorporate the data from the

secondary databases into the primary database once all the secondary database files have been transfered

to its local machine. If the data consolidation process is sucessful, both the local and remote copies of the

secondary databases will be deleted, and the bigworldSecondaryDatabase table will be cleared. If the

data consolidation process is unsuccessful, only the local copy of secondary database files are deleted.

The data consolidation tool logs to Message Logger under the ConsolidateDBs process. Under most

circumstances, the data consolidation tool will log errors from transfer_db as well. However, under some

circumstances, transfer_db errors will appear under the Tools process. The bwmachined logs, in /var/

log/messages on each machine where secondary databases are located, can help to diagnose problems,

especially those related to the failure to run transfer_db.

6.3.1. Skipping Data Consolidation

If, for some reason, there is a problem with the data consolidation tool and the server refuses to start,

the bigworldSecondaryDatabase table can be manually cleared to skip the data consolidation process.

Alternatively, consolidate_dbs can be run with the command-line parameter --clear to achieve the same

result.

While skipping the data consolidation process may allow the server to start again, the data in the secondary

databases that were not consolidated is essentially lost. Though the secondary database files still exists, it

is not recommended to try to consolidate the data after the server is restarted since the data in the primary

database may be more recent than the data in the left over secondary databases.

Ignoring SQLite Errors

6.3.2. Ignoring SQLite Errors

When consolidate_dbs encounters an error in reading a secondary database file, it will abort the entire data

consolidation process. Such errors should be investigated and, hopefully, corrected. However, if a secondary

database file is genuinely corrupted and cannot be repaired, then it may be preferable for consolidate_dbs

to read as much data from the corrupted secondary database and then proceed to consolidate the rest

of the secondary databases. When the --ignore-sqlite-errors command-line option is specified,

consolidate_dbs will proceed to consolidating the next secondary database when it encounters an error

instead of aborting the consolidation process.

Chapter 7. Controlled Startup and Shutdown

There are times when the server might need to be shut down, and restarted later in a similar state.

To tell the server to shut down in a controlled way, a message must be sent to all LoginApps.

This may be in the form of a Watcher message or a USR1 signal. The easiest way to do this is to

use the script control_cluster.py with the option stop or via WebConsole. For more details, see

“control_cluster.py” on page 67 .

Controlled startup and shutdown only work when the underlying database is MySQL — it does not work

when the XML version is used.

If the bw.xml's dbMgr/clearRecoveryData configuration option is set to false, then the server is

automatically started using information written during the controlled shutdown. If the server had failed

unexpectedly, then it is started using the disaster recovery information. Setting that configuration option to

true will cause the server to be started in its initial state, i.e., an empty space with no entities. For more

details, see “DBMgr Configuration Options” on page 35 .

The main information that is restored from the database is:

▪Spaces and their data

▪Game time

▪Which entities should be in each space.

When using MySQL as the underlying database, this information is stored in the following tables:

▪Spaces and their data — bigworldSpaces, bigworldSpaceData

▪Game time — bigworldGameTime

▪Which entities should be in each space — bigworldLogOns

For details on these tables, see the document Server Programming Guide's section MySQL Database Schema

→ Non-entity tables

For details on related scripting, see the document Server Programming Guide's chapter Controlled Startup

and Shutdown.

Chapter 8. Stress Testing with Bots

The Bots application is a process that can command arbitrary numbers of simulated clients to log into the

server and perform activities. It is written in C++, but controlled through Python, like the BigWorld Client.

8.1. The Login Process

The bots are designed to log in a server just like a real client. By default, the Bots process locates a LoginApp

by broadcasting a message requesting all LoginApps to respond, and then using the first one to reply. This

is the simplest method to log in if there is only one instance of a BigWorld server running.

The script is located in folder bigworld/bin/$MF_CONFIG, and has the following syntax:

bots [-serverName <srv> [-port <port>]]

[-username <user> [-password <pwd>] [-randomName]] [-scripts]

The options for invoking bots are described in the list below:

▪-serverName <srv>

Server that the bot clients will login to (overrides the auto-locate method).

▪-port <port>

▪-username <user>

Username that the bot clients will log in as.

▪-password <pwd>

Password for bot clients' login.

▪-randomName

Randomises the login name based on the specified <username>.

▪-scripts

Enables Bots to run scripts. This has a considerable performance cost.

The auto-locate procedure can be overridden by specifying the option -serverName <serverName> on

the command line. If both methods fail, then the Bots process requests the server name on stdin (if attached

to a terminal). If all these methods fail, then the process starts, but cannot create new bots until the server is

set (via the Python method setDefaultServer).

If the option -scripts is specified then Bots looks for the entity definition files under folder <res>/

scripts/entity_defs, and for scripts under folder <res>/scripts/bot.

Specific scripts can be created for Bots, but if a client script does not reference the client's C++ modules, then

it can be reused. Note that the bots process does not support the BigWorld Client APIs, since it is intended

to be lightweight.

In many operation scenarios the entity definition file for Bots could be either missing or different from the

definition files for the server. The Bots would use an incorrect message digest for login; and hence the login

will fail because LoginApp relies on the digest for verifying entity definition consistency. Previously, only

Watcher Interface

way around this problem is turn the digest checking on the server off in <res>/server/bw.xml's dbMgr

options section (for details check allowEmptyDigest option in “DBMgr Configuration Options” on page

35 . This option can also be changed on the fly via a watcher, using WebConsole's ClusterControl module;

see “ClusterControl” on page 45 ). However, this is not a viable option, if you want to use Bots on production

servers. You can now set custom MD5 digest in the <res>/server/bw.xml's bots section under option

loginMD5Digest. This option can also be modified on the fly through WebConsole.

Bots can now realistically simulate general Internet networking environment by introduce artificial packet

loss, network delays, sudden disconnections and frequent relogins. All this simulation can be programmed

with Bots' personality script.

8.2. Watcher Interface

It is possible to control Bots using WebConsole's ClusterControl module (for details, see “WebConsole” on

page 45 ).

The list below describes the watchers exposed by Bots:

▪command/addBots (Data type — int, Access — Write only)

Adds to the system the specified number of bots.

▪command/delBots (Data type — int, Access — Write only)

Deletes from the system the specified number of bots.

▪command/updateMovement (Data type — string, Access — Write only)

Updates the movement controllers of all bots matching the input tag, based on the current default values.

If the input tag is empty, all bots are changed.

▪defaultControllerData (Data type — string, Access — Read/Write)

The argument string that default movement controllers are to be constructed with.

▪defaultControllerType

The name of the movement controller for the bots, if that is not specified individually.

▪defaultStandinEntity

Default entity script to be used when a specific Entity type does not have its corresponding game script.

▪loginMD5Digest

32 character long MD5 digest string (in hex readable form) for server login.

▪delTaggedEntities (Data type — string, Access — Write only)

Deletes all bots matching the input tag.

▪numBots (Data type — int, Access — Read only)

Number of bots being controlled by this process.

▪pythonServerPort (Data type — int, Access — Read only)

The telnet port the process is listening on (if it was available when the application started).

▪tag (Data type — string, Access — Read/Write)

Python Interface

The current tag.

Note the unusual use of write-only fields to perform actions. The Watcher interface is rarely used, since bots

are easier to control with the script bigworld/tools/server/ bot_op.py, or the Python interface.

For more details on changing a watcher via WebConsole, see “ClusterControl” on page 45 .

8.3. Python Interface

Bots was designed to be controlled programmatically via Python, but since Python can be used interactively,

it is also a quick way to get some bots into the system and control them.

You can telnet to port 6075 to talk to the bots process, as illustrated below:

$ telnet 10.40.7.12 6075

Welcome to the Bot process

>>> BigWorld.getDefaultServer()

10.40.7.1

>>> BigWorld.setDefaultServer('server2')

>>> BigWorld.setDefaultTag('red')

>>> BigWorld.addBotsSlowly(500, 0.1)

>>> BigWorld.setDefaultTag('blue')

>>> BigWorld.addBotsSlowly(500, 0.1)

>>> BigWorld.addBotsWithName( [('Bot_01', '01'), ('Bot_02', '02')] )

Example of use of Python interface to Bots

8.3.1. Python Controller (bot_op.py)

BigWorld provides a Python program to simplify the management of large numbers of bots.

It automatically starts bot processes as needed, load balancing (based on CPU load) on all available machines

not running any other BigWorld components.

The script is located in folder bigworld/tools/server and has the following syntax:

python bot_op.py [add [<number_of_bots>]]

[del [<number_of_bots>]]

[movement <controller_type> <controller_data> [<bot_tag>]]

[set (<watcher_name> <value>)+]

[run [<command>]]

[addprocs [<num_of_procs>]]

-u [username or uid]

The options for invoking bot_op.py are described in the list below:

▪add [<number_of_bots>]

Adds the specified number of bots to the system. If <number_of_bots> is not specified, then one bots is

added.

Bots are added 16 at a time, up to a maximum of 256 per bots process.

The script waits 1 second after each add, to check it a bot was effectively added. When all bots processes

are full, it creates a new process on an available machine that is using less than 80% CPU (preferably one

that is already running some bots processes).

Methods and Attributes

101

▪addprocs [<num_of_procs>]

Spawns the given number of bots processes on each machine that is considered eligible for running them,

or 1 on each if no number is given.

▪del [<number_of_bots>]

Deletes the specified number of bots from the system. If <number_of_bots> is not specified, then it defaults

to 1. The number of bots will be ceiling clamped to the actual number of bots on the server.

▪movement <controller_type> <controller_data> [<bot_tag>]

Changes the bot's movement controller. If option <bot_tag> is specified, then only bots matching it will be

changed. Otherwise, all bots will be changed.

▪set (<watcher_name> <value>)+

Sets the watcher to specified value on all bot applications.

▪run [<command>]

Runs the specified Python command on all bots. If <command> is not specified, then it is read from stdin.

▪-u [username|uid]

Manually specify the UID of the server that the script will take action on.

8.3.2. Methods and Attributes

The list below describes the bots' attributes and methods on ClientApp and BigWorld:

▪ClientApp Attributes

▪id (int) — Bot ID (read-only).

▪spaceID(int) — ID of the space the bot is currently in. 0 means not in any space (read-only).

▪loginName(string) — Bot's login name.

▪loginPassword(string) — Bot's login password.

▪tag (string) — Bot's user-specified tag. Allows the control of partial sets of bots.

▪speed (float) — Bot's speed.

▪position (Vector3) — Bot's position.

▪yaw (float) — Bot's yaw.

▪pitch (float) — Bot's pitch.

▪roll (float) — Bot's roll.

▪isOnline(bool) — Indicates whether the bot is connected to a server (read-only).

▪isDestroyed(bool) — Indicates whether the bot has been destroyed (read-only).

▪entities(PyObject)

Dictionary-like attribute containing the list of entities that are in the simulated client's current AOI.

Methods and Attributes

▪ClientApp Methods

▪logOn()

Initiate log on process for the simulated client to connect to a BW server.

▪logOff()

Make a simulate client disconnect from server gracefully. If the simulated client is not online, it will do

nothing.

▪dropConnection()

Make a simulate client drop the connection with the server. If the simulated client is not online, it will

do nothing.

▪setConnectionLossRatio( float lossRatio )

Set up network packet loss ratio for simulating unstable network environment. Range between 0.0 and

1.0. 0.0 means no packet loss. 1.0 mean 100 percent packet loss.

▪setConnectionLatency( float minLatency, float maxLatency )

Set up simulated network data latency (in milliseconds).

▪moveTo( Math.Vector3 position )

Set next destination position for the player avatar of the simulated client.

▪snapTo( Math.Vector3 position )

Set the player avatar position for the simulated client.

▪stop()

Stop the player avatar of the simulated client moving.

▪faceTowards( Math.Vector3 direction )

Set the direction of avatar of the simulated client.

▪addTimer()

Add a timer for the bot client. This function returns an integer timer id; otherwise it returns -1.

▪delTimer( int timerID )

deletes an active timer corresponding the timerID.

▪onTick()

This optional function is invoked every game tick, should it be defined in the player Avatar script.

▪BigWorld Attrbutes

▪bots

Dictionary-like attribute containing the list of simulated clients. The key is the player's ID, and value is

an instance of ClientApp.

▪bots [<player_id>]. entities

Methods and Attributes

103

Dictionary-like attribute containing the list of entities created under the simulated client specified by

<player_id>. The key is the entity's ID, and the value is the reference to the entity.

In non-script bots, only the player entity is created and stored in this attribute.

▪bots [<player_id>].entities[<ent_id>].clientApp

Attribute referring to the ClientApp that owns the entity. An entity can access its owner ClientApp

via this attribute. BigWorld.bots[<player_id>].entities[<ent_id>].clientApp equals to

BigWorld.bots[<player_id>].

▪BigWorld Methods

▪setMovementController(string type, string data) — Context: ClientApp

Sets a new movement controller for the bot. On failure, the controller is left unchanged. Returns true

on success, false on failure.

▪setLoginMD5Digest(string MD5Digest) — Context: BigWorld

Set the 32 character long MD5Digest (in hex readable form) for server login. If the input string length is

not exactly 32 character, the MD5 digest will be reset to empty.

▪addBots( int numBots ) — Context: BigWorld

Immediately adds the specified number of simulated clients to the application. If too many bots are

added at once, then some of them may time out, depending on how long it takes to log in all of them.

▪addBotsSlowly(int numBots,float delayBetweenAdd,int groupSize=1) — Context: BigWorld

Adds bots to the system in groups of groupSize, and add a delay between each group. This method

returns immediately, adding the bots in the background, and is the best way to add bots to a system.

▪addBotsWithName( PyObject loginInfo )

Immediately add a set of bots with login name and login password specified from loginInfo. The

structure of the loginInfo is a list of tuples which consist of two strings. The first string is the login name

and the second one is login password. In the event of incorrect loginInfo, a Python exception will be

thrown.

▪delBots( int numBots) — Context: BigWorld

Deletes the specified number of simulated clients from the application. The method

onClientAppDestroy() of the personality module is called.

▪getDefaultServer() — Context: BigWorld

Returns the server name that the bots try to log in.

▪setDefaultServer( string serverName ) — Context: BigWorld

Sets the server name that the bots try to log in.

▪onTick() — Context: BWPersonality

This optional function is invoked every game tick, should it be defined in the personality module in

<res>/server/bw.xml.

▪onLoseConnection( int playerID ) — Context: BWPersonality

Controlling Movement

If this callback function is defined in the personality module specified in file <res>/server/bw.xml, it

will be invoked when a simulated client loses its connection with a server abnormally. The personality

script can decide whether the client should be destroyed or remain dormant for later reactivation by

return True or False respectively. NOTE: if this callback is not defined, bots that lose connection will be

automatically destroyed.

▪onClientAppDestroy( int playerID ) — Context: BWPersonality

If defined in the personality module specified in file <res>/server/bw.xml, this callback method is called

for each destroyed bot client. The argument is the ID of the deleted player entity. A bot client may be

destroyed due to either loss of server connection or by BigWorld.delBots() method call. If a bot client

is destroyed by BigWorld.delBots() method call, it will be disconnected from the server gracefully. We

recommend client should execute additional log off procedure in this onClientAppDestroy callback

function. An example is as following:

def onClientAppDestroyed( playerID ):

bot = BigWorld.bots[playerID]

if bot.isOnline:

bot.entities[playerID].base.logOff()

The implementation above assumes that the player entity on the server has an exposed base method

logOff(), which will then destroy the player cell and base entities on the server. The script must to be

placed under the script folder <res>/scripts/bot.

8.4. Controlling Movement

The default movement controller Patrol moves the bots along a graph read from a file.

The name of the file containing the movement graph is specified in file <res>/server/ bw.xml, on section

<bots>, by the configuration option controllerData.

<root>

...

<bots>

<controllerData> server/bots/test.bwp </controllerData>

<controllerType> Patrol </controllerType>

<password> passwd </password>

<randomName> false </randomName>

<scripts> false </scripts>

<publicKey> loginapp.pubkey </publicKey>

</bots>

...

</root>

Example file <res>/server/bw.xml

The file name might have the suffix random, in which case the start position of the bot is chosen randomly

from the set of node positions in the graph.

The format specification for the movement graph file is illustrated below:

NodeProperties Section

105

?NodeProperties

</nodeDefaults>

<nodes>

*<node>

<pos> integer integer integer </pos>

?<name> string_node_name </name>

?NodeProperties

*<edges>

<edge> string_node_name </edge>

</edges>

</node>

</nodes>

</controller>

Grammar of movement graph files

The list below describes the tags in the movement graph file:

▪edge (section nodes/node/edges)

Name of one of the nodes that this node borders. Must be the same value specified for a name section

in the file.

▪edges (section nodes/node)

Tag specifying all the nodes that this node borders.

▪name (section nodes/node)

Name used to refer to the node. If this tag is not specified for a node, then you can refer to it by an integer

sequential number, with the first node having an index of zero.

▪node (section nodes)

Tag for section specifying a specific node in the graph.

▪nodeDefaults

Tag for section specifying default values for all nodes. These values can be overridden on a per-node basis

in section nodes/node.

▪NodeProperties (section NodeDefaults and nodes/node)

For details, see “NodeProperties Section” on page 105 .

▪nodes

Tag for section specifying all the nodes in the graph.

▪pos (section nodes/node)

XYZ position of the centre of the node. The node is the area around pos, extending for n metres around

it (n being the value of NodeProperties' tag radius).

8.4.1. NodeProperties Section

The format specification for the NodeProperties section is illustrated below:

?<minStay> float </minStay>

Extending Bots

?<maxStay> float </maxStay>

?<radius> float </radius>

?<minSpeed> float </minSpeed>

?<maxSpeed> float </maxSpeed>

Grammar of NodeProperties section

The list below describes the tags in the NodeProperties section:

▪maxSpeed

Maximum speed of the bot.

▪maxStay

Maximum number of seconds that the bot should stay in the node.

▪minSpeed

Minimum speed of the bot.

▪minStay

Minimum number of seconds that the bot should stay in the node.

▪radius

Number of metres around the node's pos location to be considered as the node. When a bot is travelling

between two nodes, it chooses a random point of arrival in the destination node. This way, the bots do not

follow the exact same line. The point is chosen to lie within the radius of the destination node.

An example movement graph file is displayed below:

</nodeDefaults>

<nodes>

<node>

</node>

<node>

</node>

</nodes>

</patrolGraph>

Example movement graph file

8.5. Extending Bots

8.5.1. Creating New Movement Controllers

The default movement controller is Patrol, but you might want to create one that better represents the

expected traffic patterns and entity distributions of your game, or to stress test other aspects of the system.

Extending Bots

107

To create a new movement Controller, derive its class from the MovementController class. Also derive a class

from the MovementFactory class. The factory is used to parse the Controller data argument, and instantiate

a new movement controller, while the movement controller moves the bot around.

The fragments below list the relevant methods in the base classes:

▪Class MovementController

bool nextStep( float speed, float dTime, Vector3& position, Direction3D&

direction)

▪Class MovementFactory

MovementFactory(const char* name)

MovementController* create(

const std::string& data,

const Vector3& startPosition)

108

Chapter 9. Security

Security has been of paramount importance in the design and implementation of all parts of BigWorld.

The basic philosophy is to always handle with care any client-initiated actions or messages. This should be

accomplished in a way that does not unduly limit potential game designs.

For more details, see the document Server Programming Guide, chapter Security Systems in BigWorld

Technology.

9.1. Security of the Server

The internal network is assumed to be secure — BigWorld does not implement security measures to

safeguard processes against an attacker gaining access to the cluster's internal LAN. Operators should

ensure that the usual protections for an internal network are in place. Remote access should be very strictly

controlled.

The external points of contact are the area of most concern when running an exposed server. For BigWorld,

these are LoginApp and BaseApp, as illustrated below:

BigWorld Server components

Due to the bandwidth needs of a massively multiplayer online game, LoginApp and BaseApp are intended to

be run on machines with external access. In some sense they are the firewall. For more details, see “Blocking

Ports and Related Security Considerations” on page 110 .

Server Ports

109

LoginApp receives only fixed-length queries, making it easy and transparent to secure. This process is

expected to be tailored by customers to suit their game, but care should be taken when doing so.

BaseApp receives more complex data, including script method calls, and is the gateway to the rest of the

BigWorld Server. It has many checks in place to ensure the integrity of received data, and to discard (and

warn about) corrupted data and hacking attempts. The string CHEAT is used in the log messages when

BigWorld receives potentially malicious data that does not conform to its protocols (the CellApp may also

use this indicator.) It is advised that MessageLogger logs be monitored for messages containing that string

(for details on MessageLogger, see “MessageLogger” on page 50 ).

The security of the game-level logic rests to a certain extent with the Python scripts that implement it. For

example, an entity should not be able to stab another entity that is 100 metres away. For more details on this

topic and on server features such as physics checking, see the document Server Programming Guide.

9.2. Server Ports

The list below describes the ports used by BigWorld server:

▪20013 (Protocol: UDP, Access: External)

Used by LoginApp, this port can be overridden in the following ways:

▪In <res>/server/bw.xml file's loginApp section, set the port configuration option.

▪In the command-line arguments to LoginApp, use the -loginPort <portNum> option.

▪20017 (Protocol: UDP, Access: Internal)

The HTTP interface of Watcher. To start the interface, run bigworld/tools/server/watcher, then

connect to this port with a web browser.

▪20018 (Protocol: UDP, Access: Internal)

Used by BWMachined.

▪40001-49999 (Protocol: TCP, Access: Internal)

The Python server on BaseApp.

All BaseApps have a Python server that can be telnetted to. The port number is 40000, plus the BaseApp ID.

BaseApp ID numbers start at 1, so to talk to the third started BaseApp, telnet to 40003.

▪50001-59999 (Protocol: TCP, Access: Internal)

The Python server on CellApp.

All CellApps have a Python server that can be telnetted to. The port number is 50000, plus the CellApp

ID. CellApp ID number start at 1, so to talk to the third started CellApp, telnet to 50003.

Note

CellApp ID is not the same as the Cell ID. Cell IDs are allocated sequentially, as they

are allocated to a cluster, not when the CellApp is started.

▪32768-61000 (Protocol: TCP, Access: Internal)

Used by CellAppMgr and CellApp for viewer applications such as Space Viewer. Automatically assigned

by the kernel (see UDP entry).

Blocking Ports and Related Security Considerations

▪32768-61000 (Protocol: UDP, Access: Internal [External for BaseApp])

Used by the server components: CellApp, CellAppMgr, BaseApp, BaseAppMgr, DBMgr, MessageLogger

and StatLogger. Only the BaseApp has an external port.

Automatically assigned by the kernel, in the range of the kernel setting /proc/sys/net/ipv4/

ip_local_port_range, which defaults to 32768-61000.

The BaseApp external port may be exempted from random assignment, by specifying it in the <res>/

server/bw.xml file's baseApp section's externalPort configuration option.

The port chosen can be displayed via the Watcher interface under:

▪/(internal|external) nub/address on BaseApp.

▪/nub/address on other components.

9.3. Blocking Ports and Related Security Considerations

Since TCP/IP is not used externally, you can block all TCP traffic. Leave all UDP/IP ports 32768 and above

open, as well as the login port (20013 by default).

Use of a separate firewall machine is strongly discouraged. The BaseApp machines are designed to be the

firewalls themselves, and perform a very similar proxying function for clients. Their amount of processing

is small enough so that they can handle a whole network adapter's worth of Internet traffic. Adding another

machine would only be a waste of hardware, maintenance time, and latency. BaseApps only listen on one

UDP port — and so the whole TCP stack can be disabled on the external interface. The use of standard

firewall software such as iptables may be an appropriate way to accomplish this.

Network tools such as lsof and netstat should be consulted, to ensure that you are not running any daemons

listening on the external (or common) interface. Apart from BWMachined, you should not need to run

any daemons with BigWorld, but if you wish to, then you should ensure that their ports are blocked.

BWMachined should not listen on the external interface, only the internal interface.

Barring all TCP packets greatly improves the security of a BaseApp machine. TCP is a complicated protocol,

and requires many tables and buffers to implement or firewall. By this rationale, the security of a BaseApp

machine may be considered even better than that of an ordinary firewall, which must conditionally pass

TCP packets.

To reduce exposure to DDOS attacks, it is recommended that the BaseApp be left to choose, within the range

allocated by the operating system, a random port. This way, if an attacker discovers the IP and port of one of

the BaseApp machines, then it does not mean he will automatically know those details for the other BaseApp

machines.

111

Chapter 10. BigWorld Server Across Multiple

Machines

To start a server cluster consisting of multiple machines, you can start each process in any of the following

ways:

▪the command line

▪the control_cluster.py script

▪WebConsole's ClusterControl module

CellApps, BaseApps, and LoginApps can have multiple instances running, while the other processes can

have only one. The clients should connect to the IP address of a machine that is running a LoginApp.

10.1. How To Start

The following sub-sections describe how to start server components.

10.1.1. WebConsole

WebConsole can be used to easily start, stop and control server components.

For an outline of WebConsole, see “WebConsole” on page 45 . For operational behaviour, see the online

WebConsole documentation in the Cluster Control module.

10.1.2. Auto Configuration Via control_cluster.py

The script bigworld/tools/server/control_cluster.py is generally the easiest way to start a multi-

machine BigWorld Server. For details, see “control_cluster.py” on page 67 .

10.1.3. Hard-Coded Scripts

Create your own shell scripts using the script bigworld/tools/server/misc/startsingle.sh as a

guide.

10.1.4. Manual Start

During development and testing, it is feasible to manually start processes individually.

10.2. How To Stop

The method used to stop the system depends on the method used to start it. For example, calling the script

control_cluster.py with the option stop stops a system started with that script. Stopping LoginApp

triggers other processes to terminate, unless this feature is disabled in configuration file <res>/server/

bw.xml.

10.3. How To Monitor

Use WebConsole (for details, see “WebConsole” on page 45 .) and control_cluster.py (for details, see

“control_cluster.py” on page 67 .) for monitoring and recording real-time statistics. You can also get

profiling data from the watchers (via WebConsole's ClusterControl module). Use WebConsole's LogViewer

module to monitor the centralised log.

10.4. LoginApp and Scalability

Just as you can use multiple CellApps and BaseApps in a large cluster, you can also run multiple LoginApps

in a load sharing arrangement.

LoginApp and Scalability

To do that, DNS has to set up so that it returns multiple DNS addresses for the login server's name. If you

are using BIND, then you can simply put multiple A addresses in the zone configuration file. This means

that when a client looks up the DNS address of the login server, it receives a login server randomly selected

from the available pool.

For example, if you are using BIND as your DNS server (most Linux distributions come with BIND as their

default DNS server), the zone file can have configuration similar to the ones in the table below:

Name TTL CLASSTYPE ResourceRecord

600 IN A 10.0.0.1

600 IN A 10.0.0.2

600 IN A 10.0.0.3

Example configuration on a BIND zone file

Because there are multiple A records with different IP addresses under one name, the IP address returned

to a client when it looks up the address for the login server will be picked up from the IP address list, in a

round robin manner.

113

Chapter 11. Multiple BigWorld Servers in a Single

LAN

11.1. Keeping Processes Separate

All server components in a BigWorld server cluster run under the same UID. If you want to run multiple

server clusters on the same network (for different stages of testing, for example), then make sure that you

use different UIDs.

You also need to make sure that the LoginApps are on different machines. This is because by default each

LoginApp opens the same specific port, and therefore only one could succeed in binding to it. Alternatively,

multiple LoginApps can be run on a single machine as long as they have different port settings in <res>/

server/bw.xml.

Note

It is important the username and UID mapping is consistent across all machines on the

cluster. It is highly recommended to use centralised logins using LDAP or similar.

11.2. Centralised Cluster Monitoring

The best way to keep track of processes is via WebConsole's StatGrapher module.

The module, which can be run on any web browser, uses log files generated by StatLogger daemon to display

statistics, and has a variety of options to filter the output by user.

For more details, see “StatGrapher” on page 47 , and “StatLogger” on page 55 .

11.3. Auto-Detection of LoginApps

When running multiple servers, it is convenient for the client to auto-detect LoginApp processes (this is only

suitable for a development environment).

The LoginApps have this functionality built in, and do not require any configuration to perform it — they

details. A probe message is then sent to each server to determine who, where and what it is running. This

functionality is implemented on the client by class ServerDiscovery.

114

Chapter 12. DBMgr MySQL Support

Following is described the minimal set of steps required to enable MySQL support for a server cluster. They

assume that a MySQL database has been installed and configured as outlined in the Server Installation Guide,

section Installing the BigWorld Server, Configure MySQL server.

12.1. Compiling DBMgr with MySQL Support

The DBMgr binary provided with the default BigWorld package does not have MySQL support built-in. This

allows users to quickly start using BigWorld without having to configure a production database to test a

simple package.

First, to enable compilation of DBMgr, make sure that the MySQL development files are installed on the

build system. To do this, as root, issue:

▪On Fedora:

yum install mysql-devel

▪On Debian

apt-get install libmysql++-dev

Next, the DBMgr makefile needs to be modified, to enable MySQL support. Edit bigworld/src/server/

dbmgr/Makefile and change the variable definition from USE_MYSQL=0 to USE_MYSQL=1 (USE_XML can

be left on, without any side effects).

After these alterations, it will be possible to rebuild DBMgr. From within the bigworld/src/server/

dbmgr folder, issue the command make.

12.2. Update bw.xml To Use MySQL

Once DBMgr has been compiled to communicate with a MySQL server, the game resource configuration file

<res>/server/bw.xml needs to be updated with details on username, password and the host machine

the MySQL server is running on.

The example below illustrates FantasyDemo configuration (via fantasydemo/res/server/bw.xml):

<dbMgr>

<type> mysql </type>

<host> my_mysql_server_machine </host>

<username> bigworld </username>

<databaseName> fantasydemo </databaseName>

</dbMgr>

Example fantasydemo/res/server/bw.xml

For details on these fields and other relevant configuration options for your production environment, see

Server Configuration with bw.xml on page 6 .

12.3. Initialise Database With Entity Definitions

DBMgr requires the MySQL database table structure to exactly match the current entity definitions.

Disabling Schema-Modifying Capability

115

To initialise the MySQL database with the correct structure, go to the folder containing the server binaries

(e.g., mf/bigworld/bin/Hybrid), then run DBMgr with the appropriate argument:

$ ./dbmgr --sync-tables-to-defs

When run with the --sync-tables-to-defs option, DBMgr updates the table structure and terminates,

to allow the server to be started normally.

12.4. Disabling Schema-Modifying Capability

By default, DBMgr has the ability to alter the database schema through the --sync-tables-to-defs

command-line option or the <dbMgr>/<syncTablesToDefs> configuration option. This feature may be

undesirable in a production environment where the data in the database is highly valuable. Accidental use of

the sync-tables-to-defs feature may result in data loss. For example, if the entity definition files were deleted,

it will cause DBMgr to remove the corresponding tables in the database.

The schema-modifying capability of the DBMgr can be removed by recompiling it after setting the

variable ENABLE_TABLE_SCHEMA_ALTERATIONS to 0 in bigworld/src/server/dbmgr/Makefile.

The resulting DBMgr executable will no longer have the ability to modify the table schema. Use of the -

-sync-tables-to-defs command-line option or the <dbMgr>/<syncTablesToDefs> configuration

option will result in a warning. The DBMgr will fail to start if the database table schema does not match the

entity definitions.

The new DBMgr executable will be able to connect to a database with just the SELECT, INSERT, UPDATE

and DELETE privileges. Please note that the original DBMgr executable required the CREATE TABLE

privilege even when sync-tables-to-defs feature is not used. This is because the original DBMgr executable

automatically creates and modifies BigWorld internal tables (tables with names starting with "bigworld").

It is likely that the original DBMgr executable with the sync-tables-to-defs feature will still be useful since

manually updating the table schema to match the entity definition is a time consuming task. A copy of the

original DBMgr executable can be kept at a location where it is not likely to be invoked in the day-to-day

operations of a production server.

12.5. Enabling Secondary Databases

Secondary databases can only be enabled when MySQL support is enabled. Please see the document Server

Programming Guide's chapter Secondary Databases for details about secondary databases.

To enable secondary database support, you must:

▪Enable MySQL support (see above).

▪Build the data consolidation tool by issuing the make command from within the bigworld/src/

server/tools/consolidate_dbs directory.

▪Set the bw.xml option <baseApp>/<secondaryDB>/<enable> to true.

116

Chapter 13. RPM

BigWorld provides a RPM implementation which allows the creation of binary RPM package for

BWMachined. The RPM implementation uses the RPM system available in Linux distributions such as

Fedora to generate RPM packages. Currently, the RPM implementation only supports building RPM

packages on i386 architecture.

This chapter provides the following information:

▪Directory structure and files related to BigWorld RPM implementation.

▪How to generate BigWorld binary RPM packages.

▪Customising RPM Packages.

▪Setting up a yum repository.

▪Install, upgrade and uninstall using yum command.

▪How to obtain version number of an installed package.

13.1. Directory Structures and Files

The RPM implementation is located in the bigworld/tools/server/rpm directory:

▪The Makefile is used to generate RPM packages.

▪The generate.py is used to facilitate the creation of RPM packages.

▪The binary_rpms directory is where generated RPM packages are placed.

▪The rpm directory also contains package specific directories. For example, the bwmachined directory

contains all the files specific to BWMachined.

13.2. How to Generate Binary RPM Packages

Binary RPM packages must be generated using a normal Unix user account. Binary RPM packages must not

be generated using the root user.

Before generating RPM packages, the normal user used will need to be given read and write access to the the

directory used by the RPM system to build the RPM packages. Run the following command to determine

which directory is used by the RPM system to generate RPM packages:

$ rpm --eval %_topdir

To generate RPM package for BWMachined, go to the rpm directory and run the following command:

$ make

The generated RPM package will be placed in the binary_rpms directory.

The generated RPM package will have the following file name:

bigworld-bwmachined-<version>-<release>.i386.rpm

Customising RPM Packages

117

▪The <version> field is the version number of the BigWorld release that the package was generated from.

This field is based on the version number in bigworld/res/version.xml.

▪The <release> field identifies the specific build of this RPM package. By default, it is not used and is mapped

to patch number in bigworld/res/version.xml.

13.3. Customising RPM Packages

The most common customisation of a RPM package is the configuration file(s) installed by the package.

That is, one RPM package may include the default configuration file and another may include a customised

configuration file. These RPM packages can be distinguished by using the release field in the file name

of these packages.

For example, the configuration file included in a BWMachined RPM package is located in rpm/

bwmachined/bwmachined.conf.

To generate a BWMachined RPM package with the default configuration file and assign it the release

number 0:

▪Use the bwmachined.conf provided by default.

▪Update the Release tag in rpm/bwmachined/bwmachined_template.spec to the following:

Release: 0

▪Generate the RPM package.

The file name of the RPM package generated will be, for example, bigworld-bwmachined-1.9.1.0-

0.i386.rpm.

To generate a BWMachined RPM package with customised configuration file and assign it the release

number 1:

▪Update the bwmachined.conf as required.

▪Update the Release tag to 1.

▪Generate the RPM package.

The file name of the RPM package generated will be, for example, bigworld-bwmachined-1.9.1.0-

1.i386.rpm.

13.4. Setting up a Yum Repository

For a large environment where RPM Packages, e.g. the BWMachined RPM Package, need to be installed on

many machines, we recommend that a yum repository to be set up to provide these RPM packages. This

allows an RPM package to be installed on any machine in the cluster by running the yum command without

the need to manually copy the RPM package to each machine in the cluster.

The yum repository created will need to be made available through a web server such as the Apache HTTP

Server. The setup required is outside the scope of this document. Please consult relevant web server

documentation on how to achieve this.

Steps to set up a yum repository:

▪On the machine that will be hosting the yum repository, create a directory for the BigWorld RPM packages.

The location of this directory will depend on the web server chosen and the web server configuration

which is outside the scope of this document.

Setting up a Yum Repository

▪Copy the RPM packages to the directory created.

▪Install the createrepo package using the following command:

$ yum install createrepo

This is a utility that will generate a common metadata repository from a directory of RPM packages.

Note

You will need to be the root user to install a package.

▪Run the following command to create the necessary metadata for the yum repository:

$ createrepo <path_to_dir>

where <path_to_dir> is the path to the directory created.

For example, if the directory created is /mnt/bigworld_repo, then the command to run is the following:

$ createrepo /mnt/bigworld_repo

This creates a repodata directory in the /mnt/bigworld_repo directory.

▪The /etc/yum.conf on a machine that will access the repository created will need to be updated to

include the following setting:

[<repo_name>]

name=<repo_name>

baseurl=<url_to_repo>

enabled=1

where <repo_name> is replaced by the name given to the repository and <url_to_repo> is replaced by the

URL that refers to the repository.

Note

By default, the /etc/yum.conf contains a [main] entry. The entry above should

be added after the [main] entry.

When the yum repository is updated, for example, a new version of BWMachined RPM package is added,

then the repodata directory should be deleted and the createrepo command should be run again to

generate up-to-date metadata.

If a new RPM package was added to the repository but is not displayed on a machine when running a query

command such as

$ yum info bigworld-bwmachined

run the following command to delete the metadata used by yum to determine the remote availability of

packages:

Install, Upgrade and Uninstall using Yum Command

119

$ yum clean meatadata

When the yum command is run the next time, it will download the up-to-date metadata.

13.5. Install, Upgrade and Uninstall using Yum Command

This section describes how to install, upgrade and uninstall using the yum command. The BWMachined

package is used as an example.

13.5.1. Install and Upgrade using a RPM Package Directly

▪To install a RPM Package directly without using yum repository:

$ yum --nogpgcheck install bigworld-bwmachined-<version>-<release>.i386.rpm

where <version> and <release> are replaced by the actual version and release number.

This assumes that you are running the yum command in a directory containing the specified .rpm file.

Note

The RPM packages created by BigWorld RPM implementation are not signed, since

these packages are from trusted source. Therefore, during installation and upgrade,

GPG check is disabled.

▪To upgrade a package:

$ yum --nogpgcheck upgrade bigworld-bwmachined-<version>-<release>.i386.rpm

13.5.2. Install and Upgrade using Yum Repository

▪To install a package from a yum repository:

$ yum --nogpgcheck install bigworld-bwmachined

▪To upgrade a package:

$ yum --nogpgcheck upgrade bigworld-bwmachined

13.5.3. Remove an Installed Package

To remove an installed package:

$ yum remove bigworld-bwmachined

13.6. How to Obtain Version Number of an Installed Package

To obtain the version number of an installed package such as the BWMachined package, run the following

command:

How to Obtain Version Number of an Installed

Package

$ yum info bigworld-bwmachined

121

Chapter 14. First Aid After a Crash

If a BigWorld server component fails, then there are some steps that you should follow, to allow us to identify

and resolve the problem as quickly as possible.

The first thing you should do is determine which component crashed first. There are many ways to do this

— you can examine the logs or check the cores.

14.1. Procedure For Crash Reporting

A crash may be either intentional (an assert), or unintentional (such as segmentation fault). Intentional

crashes are often due to errors in extensions, or unexpected use of script.

When you report a crash, be sure to include in your e-mail the following information:

▪Backtrace The full call stack for the first process that crashed (use the bt command in gdb).

▪Logs 50 lines before, and 50 lines after the crash (this can be obtained by issuing the command mlcat.py

--around <corefile> --context 50), plus the last 50 lines of output generated by the process that

crashed (this can be obtained by issuing the command mlcat.py --around <corefile> --context

50 --filter pid <deadPID>).

If a more detailed investigation is required by BigWorld support staff, it is likely that it will be necessary

to provide us with the actual raw log data generated at the time of the crash. The interactive segment

selection mode of mltar.py (for details, see MessageLogger, “Command-Line Utilities” on page 52 ) is

useful for archiving the necessary files before sending them.

It is not necessary to send any core files immediately, but do keep them so we can examine them later.

It is also very helpful if you give us access via ssh to a machine where we can examine the core dump, thus

avoiding its time-consuming transmission over the Internet. This is also a good procedure because it allows

us to verify that the correct executable binary is being used, which is an especially important check if you

modified and relinked the executable, or have custom extensions, etc...

14.2. Q & A

Q: What if there is no core dump?

A: Some shells by default prevent the creation of core dumps. To change this default, before running

BigWorld processes issue the command 'ulimit -c unlimited'. This command sets resources limits

— in this case it sets the core files to unlimited.

Q: How to determine the first process that crashed?

A: This is the information that could be used to determine that:

▪Date on the core dump.

▪Log message about the unexpected death of a process.

▪'restore' messages in the backtrace.

122

Chapter 15. Common Log Messages

This section provides the description of common log messages.

15.1. Warnings

▪CellApp WARNING WatcherDoc::initWatcherDoc: unable to load

watcherdoc/cellapp.xml

This WARNING is due to an incomplete feature to add documentation to our watchers. We apologise if

it caused any confusion. You can silence this warning by creating cellapp.xml in bigworld/res/

watcherdoc. The file should contain an empty root section:

<root>

</root>

▪CellApp WARNING InterfaceElement::expandLength: Received a message

longer than normal length

We try to optimise network traffic by using the smallest integer size that can encode the expected length

of the message. For example, if we expect the message to be always less than 255 bytes, we will use just

one byte to encode the length. We use 2 bytes for messages that are expected to be less than 65535 bytes

long, etc. However, if the message exceeds the expected size, we will fallback to using 4 bytes to encode

the length. The above WARNING means that a message has exceeded the expected message length.

Unfortunately, the WARNING does not tell us which message has exceeded the message length. We have

added additional information to the WARNING in BigWorld 1.8.2.

▪ CellApp WARNING CellApp::handleGameTickTimeSlice: Last game tick took

0.21 seconds

This WARNING means that the game tick took longer than the normal 0.1 seconds. This can be due to many

reasons but usually it is due to a script function taking a long time. You should ensure that your script

code does not do anything that takes significant amount of time. You may have to break your processing

into several runs by using a timer.

▪CellApp WARNING CellApp: Scaling back!!

In response to a game tick exceeding 0.1 seconds, the CellApp will skip some of its usual processing to help

decrease the load, e.g. it will reduce the amount of updates sent to the client. This WARNING is to inform

you that clients may receive less updates e.g. entity movements may be jerky.

▪CellApp WARNING cellapp.cpp:1918: Profile CALL_TIMERS took 0.12 seconds

This means that the total time of all onTimer() callbacks in that tick took 0.12 seconds. This is bad because

one tick should be less than 0.1 seconds.

CellApp WARNING cellapp.cpp:1572: Profile GAME_TICK took 0.12 seconds

Errors

123

This means the game tick took more than 0.1 seconds. This time includes the CALL_TIMERS time above

so almost all of the game tick was in onTimer() callbacks.

▪CellApp WARNING Witness::update: 474 has a deficit of 2370 bytes (10.68

packets)

This is related to the <bitsPerSecondToClient> configuration in bw.xml. If

<bitsPerSecondToClient> is 20000, then it means that the maximum number of bytes we can send to

the client every tick is ((20000/BitsPerSecond)/UpdateHertz) - UDPOverhead = ((20000/

8)/10) - 28 = 222 bytes. The above WARNING means that we are sending 2370 bytes to the client

in one tick. This is 10.68 times the allowed data rate.

This is usually caused by method calls to the entities located on the client i.e. entities in the AoI of the player

or the player itself. If many method calls are made, or if the method calls are passing large arguments,

then we will exceed the bandwidth allocation for the client. When bandwidth allocation is exceeded, the

position updates send to the client is reduced so entity movements will become jerky.

▪CellApp WARNING controller.cpp:158: Profile SCRIPT_CALL took 0.63

seconds

CellApp WARNING Controller::standardCallback: method = onTimer; id =

17648; controllerID = 270; userArg = 260

CellApp WARNING timer_controller.cpp:156: Profile ON_TIMER took 0.62

seconds

The above 3 WARNINGs are generated together. They mean a single onTimer() callback took 0.62 seconds.

This is very bad since it means a single entity is using 6 times a normal game tick.

▪CellApp WARNING TimerController::New: Rounding up initial offset to 1

from 0 (initialOffset 0.000000)

This means that you are adding a timer that is less than 0.1 seconds long. Our minimum timeout is 1 tick.

15.2. Errors

▪CellApp ERROR CellApp: Scale back exhausted. spareTime 0.016686

tickPeriod 0.124365, real/entities 887/887, recvs 4504 trys 3958 good 0

errs (546 sels), nub timers 753 calls 752 rescheds

The means that the game tick has exceeded 0.1 seconds for many ticks. The CellApp has already reduced

updates to the clients to a minimum but still cannot keep the game tick shorter than 0.1 seconds.

▪CellApp ERROR Chunk::load: Failed to load chunk

001xfffx/sep/0010fffco: unknown error in item 'model'

This means that there is an error loading a model in the chunk 0010fffco. Unfortunately, there is not enough

information to identify what is causing the error. You can try loading the space in World Editor and see if

the same error occurs. World Editor may be able to provide more information about the error.

Errors

▪CellApp ERROR Received message id 41 for nonexistent entity 2844 from

192.168.50.21:49140

Message 41 is the destroyEntity message. This means that entity 2844 was already destroyed when

the CellApp received this message. Make sure that you have not called "self.destroy()" in the cell and

"self.destroyCellEntity()" in the base at the same time.

This may also be an indication of a bug in BigWorld. If you are not able to resolve the issue, please send

us more log output around the time of this error.

▪CellApp ERROR DirMappingLoader::stepLoad: Stepping a loaded column

CellApp ERROR DirMappingLoader::stepUnload: Unloading UNLOADED column

Please ignore the above ERROR messages. They are not errors. We have removed these error messages in

BigWorld 1.8.2.

▪BaseApp ERROR BaseApp::setClient: Could not find base id 2881

BaseApp ERROR getProxyForCall: No proxy set!

The above messages means that the entity (2881) has been destroyed but is still receiving messages. This

may be due to another entity calling a method using a mailbox of the destroyed entity. You have to

make sure that when an entity is destroyed, all other entities holding a mailbox to the destroyed entity is

informed so that they can reset the mailbox.

This may also be an indication of a bug in BigWorld. If you are unable to resolve the issue please send us

more logs around the time of this error.

▪BaseApp ERROR 0.0.0.0:0: Exception was thrown: REASON_GENERAL_NETWORK

This means that there was an error with the network. Either the machine has been disconnected from the

network, or there is some sort of hardware error.

▪BaseApp ERROR 10.40.3.17:56859: Exception was thrown:

REASON_NO_SUCH_PORT

This means that either the machine has been disconnected from the network, or the destination process

has crashed.

125

Chapter 16. Clock

The BigWorld server relies heavily on having an accurate clock for timers and load calculation. Unstable

clocks (clocks that may move backward) can cause undesired behaviour and even crashes. Therefore properly

configuring how BigWorld reads the time is essential for a stable, efficient server.

16.1. BigWorld Timing Methods

BigWorld provides three timing methods to select between. This can be configured by editing the file /etc/

bwmachined.conf For example, to use the gettime timing method, include a section like this:

[TimingMethod]

gettime

▪rdtsc reads the time stamp counter register on the CPU. It is by far the fastest and least stable method

of deducing the current time. It is known to become unstable in multi-cored machines where a process

moves between cores or if frequency scaling is used. In many systems however is completely stable and

could therefore be used.

▪gettime uses the kernel's clock driver to access the current time. It is far slower than rdtsc but has the

ability to draw on multiple time sources to ensure a good compromise between speed, accuracy and

stability is reachable. For additional system level configuration see: “Linux Clock Source” on page 125

▪gettimeofday is a deprecated timing method. It is similar to gettime but has a lower theoretical

maximum resolution and may be disrupted by NTP.

16.2. Linux Clock Source

Linux should automatically select an appropriate clock driver for its internal timekeeping, however at times

the sysadmin may need to select a different one. If gettime is used as the timing method this will have a

direct impact to how BigWorld behaves.

A user may check which clocksource they are using with the following command run as root:

#cat /sys/devices/system/clocksource/clocksource0/current_clocksource

You may check which options available with:

#cat /sys/devices/system/clocksource/clocksource0/available_clocksource

tsc and jiffies are known to be sometimes unstable and acpi_pm will also be unstable with some

motherboard chipsets. We suggest you use hpet if it is available and if it isn't you should try acpi_pm.

Server Operations Guide

server_operations_guide

Navigation menu

Versions of this User Manual:

Views

Navigation