Server Operations Guide

server_operations_guide

User Manual: Pdf

Open the PDF directly: View PDF .
Page Count: 125

Download
Open PDF In Browser	View PDF

Server Operations Guide

BigWorld Technology 1.9.1. Released 2008.
Software designed and built in Australia by BigWorld.
Level 3, 431 Glebe Point Road
Glebe NSW 2037, Australia
www.bigworldtech.com
Copyright © 1999-2008 BigWorld Pty Ltd. All rights reserved.
This document is proprietary commercial in confidence and access is restricted to authorised users. This document is
protected by copyright laws of Australia, other countries and international treaties. Unauthorised use, reproduction or
distribution of this document, or any portion of this document, may result in the imposition of civil and criminal penalties as
provided by law.

Table of Contents
1. Introduction .................................................................................................................................. 5
2. Server Configuration with bw.xml ................................................................................................ 6
2.1. The entry parentFile ....................................................................................................... 6
2.2. User dependent configuration ............................................................................................. 7
2.3. Command-Line Options ...................................................................................................... 7
2.4. General Configuration Options ........................................................................................... 7
2.5. Network Configuration Options ........................................................................................ 13
2.6. Load Balancing Configuration Options .............................................................................. 14
2.6.1. Entities Cardinality Configuration Options ............................................................. 14
2.7. BaseApp Configuration Options ........................................................................................ 15
2.7.1. Secondary Database Configuration Options ............................................................ 20
2.7.2. Packet Log Configuration Options .......................................................................... 20
2.7.3. ID Configuration Options ....................................................................................... 21
2.7.4. Client Upstream Limits ........................................................................................... 21
2.8. BaseAppMgr Configuration Options ................................................................................. 22
2.9. Bots Configuration Options ............................................................................................... 23
2.10. CellApp Configuration Options ....................................................................................... 25
2.10.1. Noise Configuration Options ................................................................................ 30
2.10.2. ID Configuration Options ..................................................................................... 30
2.10.3. CellApp Profiles Configuration Options ................................................................ 31
2.11. CellAppMgr Configuration Options ................................................................................ 32
2.12. DBMgr Configuration Options ........................................................................................ 35
2.12.1. Data Consolidation Options .................................................................................. 39
2.13. LoginApp Configuration Options .................................................................................... 39
2.14. Reviver Configuration Options ........................................................................................ 42
2.14.1. Reviver's BaseAppMgr Configuration Options ...................................................... 43
2.14.2. Reviver's CellAppMgr Configuration Options ....................................................... 43
2.14.3. Reviver's DBMgr Configuration Options ............................................................... 43
2.14.4. Reviver's LoginApp Configuration Options ........................................................... 44
3. Admin Tools ................................................................................................................................ 45
3.1. WebConsole ...................................................................................................................... 45
3.1.1. Modules ................................................................................................................. 45
3.1.2. Configuration ......................................................................................................... 49
3.1.3. How to Start ........................................................................................................... 49
3.2. Logger Daemons ............................................................................................................... 50
3.2.1. MessageLogger ....................................................................................................... 50
3.2.2. StatLogger .............................................................................................................. 55
3.3. Server Command-Line Utilities ......................................................................................... 67
3.3.1. control_cluster.py ........................................................................................... 67
3.3.2. eload (Entity Loader) ............................................................................................. 69
3.3.3. MessageLogger Related Utilities ............................................................................. 69
3.3.4. mls (Machine List) ................................................................................................. 70
3.3.5. runscript ............................................................................................................. 70
3.4. Standard GUI Applications ............................................................................................... 70
3.4.1. BWPanel ................................................................................................................. 71
3.4.2. Space Viewer .......................................................................................................... 75
4. Watcher (Web Interface) ............................................................................................................... 84
5. Fault Tolerance ............................................................................................................................ 86
5.1. CellApp Fault Tolerance .................................................................................................... 86
5.2. BaseApp Fault Tolerance ................................................................................................... 86
5.2.1. Distributed BaseApp Backup Method ..................................................................... 87
5.2.2. Non-Distributed BaseApp Backup Method .............................................................. 87
5.2.3. IP Address Change ................................................................................................. 89
5.2.4. Further Considerations ........................................................................................... 89

5.3. Fault Tolerance with Reviver ............................................................................................. 90
5.3.1. Specifying Components to Support ......................................................................... 90
5.3.2. Command-Line Options .......................................................................................... 91
6. Backups and Disaster Recovery ................................................................................................... 92
6.1. Disaster Recovery .............................................................................................................. 92
6.2. Database Snapshot Tool .................................................................................................... 92
6.2.1. Configuration ......................................................................................................... 93
6.3. Data Consolidation Tool .................................................................................................... 95
6.3.1. Skipping Data Consolidation .................................................................................. 95
6.3.2. Ignoring SQLite Errors ............................................................................................ 96
7. Controlled Startup and Shutdown ............................................................................................... 97
8. Stress Testing with Bots ............................................................................................................... 98
8.1. The Login Process ............................................................................................................. 98
8.2. Watcher Interface .............................................................................................................. 99
8.3. Python Interface .............................................................................................................. 100
8.3.1. Python Controller (bot_op.py) ............................................................................ 100
8.3.2. Methods and Attributes ........................................................................................ 101
8.4. Controlling Movement .................................................................................................... 104
8.4.1. NodeProperties Section ......................................................................................... 105
8.5. Extending Bots ................................................................................................................ 106
8.5.1. Creating New Movement Controllers .................................................................... 106
9. Security ...................................................................................................................................... 108
9.1. Security of the Server ...................................................................................................... 108
9.2. Server Ports ..................................................................................................................... 109
9.3. Blocking Ports and Related Security Considerations ......................................................... 110
10. BigWorld Server Across Multiple Machines .............................................................................. 111
10.1. How To Start ................................................................................................................. 111
10.1.1. WebConsole ........................................................................................................ 111
10.1.2. Auto Configuration Via control_cluster.py ................................................... 111
10.1.3. Hard-Coded Scripts ............................................................................................. 111
10.1.4. Manual Start ....................................................................................................... 111
10.2. How To Stop ................................................................................................................. 111
10.3. How To Monitor ............................................................................................................ 111
10.4. LoginApp and Scalability .............................................................................................. 111
11. Multiple BigWorld Servers in a Single LAN ............................................................................. 113
11.1. Keeping Processes Separate ............................................................................................ 113
11.2. Centralised Cluster Monitoring ...................................................................................... 113
11.3. Auto-Detection of LoginApps ........................................................................................ 113
12. DBMgr MySQL Support ........................................................................................................... 114
12.1. Compiling DBMgr with MySQL Support ....................................................................... 114
12.2. Update bw.xml To Use MySQL ..................................................................................... 114
12.3. Initialise Database With Entity Definitions .................................................................... 114
12.4. Disabling Schema-Modifying Capability ........................................................................ 115
12.5. Enabling Secondary Databases ....................................................................................... 115
13. RPM ......................................................................................................................................... 116
13.1. Directory Structures and Files ........................................................................................ 116
13.2. How to Generate Binary RPM Packages ......................................................................... 116
13.3. Customising RPM Packages ........................................................................................... 117
13.4. Setting up a Yum Repository .......................................................................................... 117
13.5. Install, Upgrade and Uninstall using Yum Command ..................................................... 119
13.5.1. Install and Upgrade using a RPM Package Directly .............................................. 119
13.5.2. Install and Upgrade using Yum Repository .......................................................... 119
13.5.3. Remove an Installed Package ............................................................................... 119
13.6. How to Obtain Version Number of an Installed Package ................................................. 119
14. First Aid After a Crash ............................................................................................................. 121
14.1. Procedure For Crash Reporting ...................................................................................... 121

iii

14.2. Q & A ...........................................................................................................................
15. Common Log Messages ............................................................................................................
15.1. Warnings .......................................................................................................................
15.2. Errors .............................................................................................................................
16. Clock ........................................................................................................................................
16.1. BigWorld Timing Methods .............................................................................................
16.2. Linux Clock Source ........................................................................................................

121
122
122
123
125
125
125

Chapter 1. Introduction
BigWorld Technology is BigWorld's middleware for implementing Massively Multiplayer Online Games.
This document is a guide to performing operations with the server software. It is not intended for game
designers or game logic implementers, but rather for 'machine room' or 'cluster control' operators and
administrators.
It is assumed that the server has been installed according to the instructions in the document Server
Installation Guide. An understanding of the basic BigWorld processes is also assumed. For more details on
these processes, see the document Server Overview's chapters Design Introduction and Server Components.

Note
For details on BigWorld terminology, see the document Glossary of Terms.

Chapter 2. Server Configuration with bw.xml
The single most important configuration file on the server is /server/bw.xml, where is the
resource tree used by the server (usually specified in ~/.bwmachined.conf).
All sever processes read this file. It contains many parameters, all of which are described in this chapter. The
default values are appropriate for many different games, and care should be taken when changing them,
since it might affect performance.
On the description of the parameters, please note the following:
▪ Boolean parameters should be specified as true or false.
▪ Where a tag is specified as tag1/tag2, the second tag is specified inside the scope of the first one. For
example, the tag dbMgr/allowEmptyDigest is specified as:

true

2.1. The entry parentFile
The entry parentFile in the configuration file specifies the next file in the chain of files where BigWorld
should look for an entry for a configuration option.
To assign a default value to a configuration option, BigWorld follows the steps below:
▪ Searches for an entry for the configuration option in file /server/bw.xml.
▪ If the file does not contain an entry for the configuration option, then the chain of configuration files is
inspected, until the entry is found.
▪ If the entry is not specified in any of these files, a hard-coded default is used.
All default values are stored in file bigworld/res/server/defaults.xml. Typically, the entry
parentFile in file /server/bw.xml is set to server/defaults.xml, and only non-default
options are stored in your server/bw.xml.
The example below shows the configuration option maxCPUOffload in section balance having its default
value overridden:

User dependent configuration

Overriding default values for configuration options

2.2. User dependent configuration
A file that is user dependent can be used instead of bw.xml. This is useful to allow multiple users to run
from the same resource tree. In a production environment, for example, you may run the resources using a
test user before using the production user.
If a file with the name server/bw_.xml exists, this is used as the start of the server
configuration chain instead of server/bw.xml.
Typically, the parentFile section in this file would refer to server/bw.xml and only options specific to
the user, such as dbMgr/databaseName would be in this file.

2.3. Command-Line Options
The configuration options specified in file /server/bw.xml can also be overridden via commandline arguments.
To override a default value, add arguments in the format +optionName value.
The example shows the baseApp section's configuration option pythonPort having its default value
changed to 40001, and the option archivePeriod changed to 0:
baseapp +baseApp/pythonPort 40001 +baseApp/archivePeriod 0

Values changed via the command line are not sent to components started via BWMachined. This includes
using WebConsole, control_cluster.py, and components started by a Reviver process.

2.4. General Configuration Options
The list below describes the general configuration options:
▪ bitsPerSecondToClient (Integer)
Desired default bandwidth from server to the client. To calculate the number of bytes to be sent in each
packet, the formula below is used (where UDP_OVERHEAD is 28 bytes):
packetSize = (bitsPerSecondToClient / 8 / gameUpdateHertz) - UDP_OVERHEAD
▪ debugConfigOptions (Integer)
Level of logging information generated when processing configuration parameters in file /
server/ bw.xml.
The possible values are described in the list below:
▪ 0
No log is generated.
▪ 1
A log message is generated for each configuration option read.
▪ 2

General Configuration Options
A verbose message is generated for each configuration option read.
▪ desiredBaseApps (Integer)
Number of BaseApps that need to be running for the server to start.
▪ desiredCellApps (Integer)
Number of CellApps that need to be running for the server to start.
▪ externalInterface (String)
Network adapter interface to use for external communication, if not explicitly set by the server component.
In a production environment, BaseApps are recommended to have two Ethernet adapters: one adapter
connected to the Internet, and a separate one connected to the internal LAN.
During development, there is no problem with using the same interface.
Accepted formats are:
▪ Adapter name ‐ Examples: eth0, eth1
▪ IP[/netmask] ‐ Examples: 10.5.2.1, 10.0.0.0/8, 192.168.5.0/24
▪ Domain name ‐ Examples: intern0.cluster/24, extern5.cluster/24
This value can be overridden by a tag with same name in the following sections:
▪ baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .
▪ loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .
▪ externalLatencyMax (Float)
Maximum number of seconds by which packets sent from the server process will be artificially delayed.
Each packet will be randomly delayed between this value and externalLatencyMin.
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .
▪ loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .
This feature is useful for testing during development.
See also ??? on page 9 , ??? on page 8
▪ externalLatencyMin (Float)
Minimum number of seconds by which packets sent from the server process will be artificially delayed.
Each packet will be randomly delayed between this value and externalLatencyMax.
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .
▪ loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .
This feature is useful for testing during development.

General Configuration Options
See also ??? on page 9 , ??? on page 8
▪ externalLossRatio (Float)
Proportion of outgoing packets that will be dropped by this processes external nub to simulate loss on
the external network. This is a value between 0.0 and 1.0 indicating what proportion of packets will be
dropped.
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For more details, see “BaseApp Configuration Options” on page 15 .
▪ loginApp ‐ For more details, see “LoginApp Configuration Options” on page 39 .
This feature is useful for testing during development.
See also ??? on page 8 , ??? on page 8
▪ gameUpdateHertz (Integer)
Number of times per second that the server should send an update to the clients. This corresponds to the
game tick frequency.
▪ hasDevelopmentAssertions (Boolean)
Flag indicating whether server should be aggressive in its use of assertions.
This option should be set to true during development, and to false when running a production server.
For example, if this option is set to true, then a corrupted packet sent from a client can cause an assertion
on the server, while if this is set to false, then only an error message is generated and the server component
continues to run.
▪ internalInterface (String)
This tag is deprecated, and its use is not recommended. For details, see the document Server Overview's section
Server Components → BWMachined → BWMachined interface discovery.
Network adapter interface to use for internal communication, if not explicitly set by the server component.
Accepted formats are:
▪ Adapter name ‐ Examples: eth0, eth1
▪ IP[/netmask] ‐ Examples: 10.5.2.1, 10.0.0.0/8, 192.168.5.0/24
▪ Domain name ‐ Examples: intern0.clust/24, extern5.clust/24
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For details, see “BaseApp Configuration Options” on page 15 .
▪ baseAppMgr ‐ For details, see “BaseAppMgr Configuration Options” on page 22 .
▪ cellApp ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ cellAppMgr ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ dbMgr ‐ For details, see “DBMgr Configuration Options” on page 35 .
9

General Configuration Options
▪ loginApp ‐ For details, see “LoginApp Configuration Options” on page 39 .
▪ internalLatencyMax (Float)
Maximum number of seconds by which packets sent from the application's internal nub will be delayed.
For more details, see internalLatencyMin.
See also internalLossRatio.
▪ internalLatencyMin (Float)
Minimum number of seconds by which packets sent from the application's internal nub will be delayed.
Each packet will be randomly delayed between this value and internalLatencyMax.
This feature is useful for testing during development.
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For details, see “BaseApp Configuration Options” on page 15 .
▪ baseAppMgr ‐ For details, see “BaseAppMgr Configuration Options” on page 22 .
▪ cellApp ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ cellAppMgr ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ dbMgr ‐ For details, see “DBMgr Configuration Options” on page 35 .
▪ loginApp ‐ For details, see “LoginApp Configuration Options” on page 39 .
▪ reviver ‐ For details, see “Reviver Configuration Options” on page 42 .
See also options internalLatencyMax and internalLossRatio.
▪ internalLossRatio (Float)
Proportion of packets on this application's internal nub that will be dropped to simulate loss on the internal
network. This is a ratio between 0.0 and 1.0.
This feature is useful for testing during development.
This value can be overridden by a tag with the same name in the following sections:
▪ baseApp ‐ For details, see “BaseApp Configuration Options” on page 15 .
▪ baseAppMgr ‐ For details, see “BaseAppMgr Configuration Options” on page 22 .
▪ cellApp ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ cellAppMgr ‐ For details, see “CellApp Configuration Options” on page 25 .
▪ dbMgr ‐ For details, see “DBMgr Configuration Options” on page 35 .
▪ loginApp ‐ For details, see “LoginApp Configuration Options” on page 39 .
▪ reviver ‐ For details, see “Reviver Configuration Options” on page 42 .
See also options internalLatencyMax and internalLatencyMin.

General Configuration Options
▪ loggerID (Integer)
The ID used by the process when registering with MessageLoggers. If this ID does not match a
MessageLogger's filter, the process will not log to that MessageLogger. (For details on MessageLogger, see
“MessageLogger” on page 50 ).
The value range of the ID is 0-255.
Multiple BigWorld servers can share the same logger process. If this behaviour is not desired, then you
can use a unique loggerID per server instance ‐ this will cause MessageLogger to filter out all messages
that do not match the loggerID it has been told to monitor.
▪ logSpamPatterns (List of Strings)
A list of log message prefixes can be specified which will be suppressed on a per-second basis if the number
sent to MessageLogger exceeds a certain threshold. Note that this is not intended a mechanism to sweep
error messages "under the carpet"; it is designed to reduce the network load that can be generated by log
traffic, which tests have indicated can be in excess of actual game traffic in some situations if suppression
is disabled.
This option can be overridden by a tag with the same name in any app section. Note that the overriding
does not merge the suppression lists, it simply replaces the global list with the one defined at the app level.
Additionally, the list of suppression patterns can be modified at runtime using the logger/
addSpamSuppressionPattern and logger/delSpamSuppressionPattern watchers.
Please see bigworld/res/server/defaults.xml for an example of a suppression list.
▪ logSpamThreshold (Integer)
The number of a particular log message that can be sent to MessageLogger in a single second before
suppression will take place. Note that only messages matching one of the (see
above) will be suppressed.
▪ monitoringInterface (String)
Network adapter interface to use for non-game communications, such as logging output, telnet sessions,
and watcher requests. For example: eth0.
See the BaseApp's configuration option externalInterface for accepted formats.
▪ numStartupRetries (Integer)
Number of times that CellApps and BaseApps will try to locate other components when starting up.
Each attempt is one second apart, so this value roughly indicates the number of seconds that these two
components can be started before the other 'global' server components have started.
▪ outputFilterThreshold (Integer)
Value used to filter the messages that are printed and sent to the logger.
All messages are tagged with an integer value. If the message number is greater than or equal to the filter
value, then the message is allowed (the bigger the value, the more messages are filtered out).
For example, a threshold of 2 allows only INFO messages and higher (TRACE and DEBUG messages are
filtered out).
The possible values and their message thresholds are described in the list below:
▪ 0 ‐ MESSAGE_PRIORITY_TRACE

General Configuration Options
▪ 1 ‐ MESSAGE_PRIORITY_DEBUG
▪ 2 ‐ MESSAGE_PRIORITY_INFO
▪ 3 ‐ MESSAGE_PRIORITY_NOTICE
▪ 4 ‐ MESSAGE_PRIORITY_WARNING
▪ 5 ‐ MESSAGE_PRIORITY_ERROR
▪ 6 ‐ MESSAGE_PRIORITY_CRITICAL
▪ 7 ‐ MESSAGE_PRIORITY_HACK
▪ 8 ‐ MESSAGE_PRIORITY_SCRIPT
▪ personality (String)
Name of the personality module for the server.
This module should contain things such as methods to be called back from the server (for example, when
the server is ready). The personality module is usually named after your game.
If not specified, the module named BWPersonality is used.
▪ production (Boolean)
If set to true, enables the server processes to run in a production mode which makes a best attempt
at emitting ERROR messages when encountering configuration settings that are considered detrimental
for a production environment. In rare cases this may also prevent server processes from starting if the
configuration options are seen to be completely unrealistic for a production environment.
Currently this is a global configuration option and cannot be set per server application type.
▪ shutDownServerOnBadState (Boolean)
Flag indicating whether server should be shut down when in an unrecoverable state.
Currently, the following scenarios are handled:
▪ All CellApps are dead.
▪ Currently, the following scenarios are handled:
▪ All BaseApps are dead.
▪ Old BaseApp fault tolerance style is being used, and there are no backup BaseApps.
▪ Two BaseApps die in quick succession, and thus base entities originally residing on the first BaseApp
are not restored from the second one to a third one, and are therefore lost forever.
▪ shutDownServerOnBaseAppDeath (Boolean)
If set to true, the entire server will be shut down if a single BaseApp dies. Normally, the fault tolerance
system would allow the server to continue running.
▪ shutDownServerOnCellAppDeath (Boolean)
If set to true, the entire server will be shut down if a single CellApp dies. Normally, the fault tolerance
system would allow the server to continue running.

Network Configuration Options
▪ shouldUseChecksums (Boolean)
If set to true, then all packets sent between server components will be checksummed to verify their
correctness. This is in addition to the UDP checksum automatically provided by the Linux kernel and
protects against packet corruption by buggy network drivers. If a corrupted packet is detected by Mercury
(meaning that it has somehow passed the UDP checksum), you will see an error message like:
ERROR Nub::processFilteredPacket( 10.40.9.103:32823 ): Packet (flags 178, size
1459) failed checksum (wanted 3dc56738, got 9fe7000a)
If after running servers for long enough and not seeing this error message you feel confident that the
UDP checksum is reliable enough on your hardware, you can disable this option for a small performance
improvement. The checksum is very simple and fast to calculate so this is likely to have only a small impact
on performance.
▪ shuttingDownDelay (Float)
Number of seconds that the server should wait before a requested controlled shutdown is actually
performed.
▪ useDefaultSpace (Boolean)
Flag indicating whether to automatically create an initial space when he server starts.
This option is ignored if spaces are loaded from the database during start-up.

2.5. Network Configuration Options
The configuration options specified in this section relate to network communication and the behaviour of
various aspects of communication channels.
The options specified in the following list are specifically related to the behaviour of channels when packets
start overflowing. This can occur when the send window fills up and buffering of packets is required in order
to handle packet resends.
The maximum packet options defines a per channel type threshold to assist in preventing channels from
using indefinate amounts of memory while buffering overflow packets.
▪ maxChannelOverflow/isAssert (Boolean)
Specifies whether the offending channel should raise a program assertion, effectively terminating the
process, when the maximum number of overflow packets has been reached.
▪ maxChannelOverflow/external (Integer)
Number of packets to allow to overflow on an external channel before raising an ERROR message, or
ASSERT'ing if has been set to true. A value of 0 disables any log messages and assertions
from occuring.
▪ maxChannelOverflow/internal (Integer)
Number of packets to allow to overflow on an internal channel before raising an ERROR message, or
ASSERT'ing if has been set to true. A value of 0 disables any log messages and assertions
from occuring.
▪ maxChannelOverflow/indexed (Integer)
Number of packets to allow to overflow on an indexed channel before raising an ERROR message, or
ASSERT'ing if has been set to true. A value of 0 disables any log messages and assertions
from occuring.

Load Balancing Configuration Options
An indexed channel is a channel that is used for communicating directly to an entity.

2.6. Load Balancing Configuration Options
The load balancing configuration options are specified in file /server/bw.xml under the section
, and are described below:
▪ maxCPUOffload (Float)
Estimated amount of CPU processing that can be offloaded from a cell to another in one tick of load
balancing.
A larger value should result in faster changes to the server's load balancing. This value is a fraction of 100%
CPU usage, and its range is from 0.0 through 1.0, but is likely to always be less than 0.1.
▪ minEntityOffload (Integer)
Using the option maxCPUOffload, the load balancing algorithm may decide that less than one entity needs
to be offloaded. Since this can get rounded down to zero, the load balancing can get restricted when small
adjustments need to be made.
This option set a minimum value for this, so if it's set to 1 (the default), then the load balancing will have
the capacity to make small adjustments.
▪ minMovement (Float)
Minimum number of metres that a partition line should move when load balancing.
A non-zero value can be useful to avoid the load balancing getting stuck.
▪ slowApproachFactor (Float)
Aggressiveness of the load balancing when cells are close to being balanced.
The value range is from 0 through 1.
A smaller value will result in a slower approach to being balanced. Too high a value may result in some
instability when the space is close to being balanced.

2.6.1. Entities Cardinality Configuration Options
The configuration options for load balancing based on the number of entities are specified in file /
server/bw.xml under section /, and are described below:
▪ enable (Boolean)
Flag indicating whether the number of entities should be used to calculate load, rather than CPU load.
In normal situations, the server uses the CPU load on the CellApps to load balance. But sometimes it is
desirable to use the number of entities per CellApp instead.
This may be useful, for example, when running multiple CellApps on a single machine for testing.
See also option demo/numEntitiesPerCell.
▪ numEntitiesPerCell (Float)
If option demo/enable is true, then this option is used for calculating a CellApp's load.

BaseApp Configuration Options
The load is calculated as numEntities / numEntitiesPerCell.
See also option demo/enable.

2.7. BaseApp Configuration Options
The BaseApp configuration options are specified in file /server/bw.xml under the section
, and are described below:
▪ archivePeriod (Float)
Period length in seconds where each entity is written to the database for the purpose of disaster recovery.
Each Entity is guaranteed to have an archive less than 2 x archivePeriod old. Setting it to zero switches
off archiving.
A large value increases performance, but reduces the effectiveness of eventual disaster recovery. The
opposite is true for a small value.
If secondary databases are disabled, this configuration option controls how often the entity is written to
the primary database. In this case, this configuration option has a dramatic impact on the performance of
the primary database when there are a large number of BaseApps. It is recommended to start with large
values (a few minutes), and perform database testing and tuning before reducing it.
If secondary databases are enabled, this configuration option controls how often the entity is written to the
secondary database. In this case, this configuration option can be set to a relatively small value (less than
a minute) since it only impacts the BaseApp machine and the load is independent of number of BaseApps
- unless the secondary database directory is on a shared network drive. For more details on secondary
databases, see Server Programming Guide's chapter Secondary Databases.
This option is also available for CellAppMgr.
▪ backupPeriod (Float)
Number of seconds between backups of each base entity to its backup BaseApp. This value is rounded
to the nearest game tick.
As a first level of fault tolerance, base entities can be copied to a backup BaseApp (i.e., backup to RAM),
while cell entities are copied to their base entity. For more details on BaseApp and CellApp fault tolerance,
see the document Server Programming Guide's chapter Fault Tolerance.
The value for this option is very dependant on the game. A small value means frequent backups, and
consequently less lost data in case a BaseApp fails. But backups cost bandwidth and CPU on the BaseApp.
In general this period can be much smaller than the one specified in option archivePeriod.
See also option backupPeriod on “CellApp Configuration Options” on page 25 .
▪ backupTimeout (Float)
Number of seconds for a BaseApp to respond to the watchdog messages from its backup BaseApp before
it is considered dead and the backup BaseApp takes its place.
▪ backUpUndefinedProperties (Boolean)
Flag indicating whether undefined properties should be backed up.
Properties of an entity are defined in the entity's definition file. However, it is possible to define additional
properties for this entity in the base script of this entity. For example, an additional property can be

BaseApp Configuration Options
defined by initialising it in the constructor of this entity class. These additional properties are referred to
as undefined properties.
If this option is set to true, undefined properties will be backed up and an error will be emitted for each of
the properties that cannot be pickled. If this option is set to false, undefined properties will not be backed
up. Default value is true.
▪ clientOverflowLimit (Integer)
If the send window for the channel to the client grows larger than this many packets, the client is
disconnected.
Generally, it is better to rely on inactivityTimeout to detect an unresponsive client and so this option
should be set to greater than inactivityTimeout * gameUpdateHertz.
See also option inactivityTimeout on “BaseApp Configuration Options” on page 15
▪ createBaseElsewhereThreshold (Float)
Threshold of local BaseApp load below which calls to BigWorld.createBaseAnywhere cause the new
base entity to be created locally.
▪ externalInterface
For details, see “General Configuration Options” on page 7 .
▪ externalLatencyMax
For details, see “General Configuration Options” on page 7 .
▪ externalLatencyMin
For details, see “General Configuration Options” on page 7 .
▪ externalLossRatio
For details, see “General Configuration Options” on page 7 .
▪ externalPort (Integer)
Port that may be used for the BaseApp's external socket ‐ the BaseApp can have more than one
externalPort definition.
This option is useful when BaseApp is running behind a firewall and NAT port mappings need to be set up.
If the BaseApps are run behind a firewall, then each BaseApp expected to run on a single machine should
have an externalPort definition.
If this option is not specified, or all specified externalPorts are taken. then the BaseApp will bind to any
available port on the external interface.
▪ inactivityTimeout (Integer)
Number of seconds that a proxy will proceed without communication from the client before it considers
the connection to be dead.
See also option clientOverflowLimit on “BaseApp Configuration Options” on page 15
▪ internalInterface
For details, see “General Configuration Options” on page 7 .

BaseApp Configuration Options
This tag is deprecated, and its use is not recommended. For details, see the document Server Overview's section
Server Components → BWMachined → BWMachined interface discovery.
▪ internalLatencyMax
For details, see “General Configuration Options” on page 7 .
▪ internalLatencyMin
For details, see “General Configuration Options” on page 7 .
▪ internalLossRatio
For details, see “General Configuration Options” on page 7 .
▪ loadSmoothingBias (Float)
Value to smooth the load changes on a component by when calculating new load.
The BigWorld server uses the load on a component to perform its load balancing. Unfiltered, the load can
change too quickly to be useful. The option loadSmoothingBias is used to smooth out this value.
The filtered value is calculated at each game tick as follows:
newSmoothedLoad = (1 - loadSmoothingBias) * oldSmoothedLoad + loadSmoothingBias
* load
This option is also available for CellApps and CellAppMgr.
▪ minClientInactivityResendDelay (Float)
Minimum number of seconds that the BaseApp waits before resending unacknowledged packets to the
client. BigWorld also uses other metrics like the round-trip-time to the client to adjust the resend timeout
period. However, BigWorld will not set the resend timeout period to less than the value specified in this
configuration option.
The BaseApp will resend packets to the client due various reasons like lost packets, network jitter or client
delays in sending packet acknowledgements. Increasing the minimum resend delay will generally reduce
the number of packet resends. However, increasing this value will also increase the impact of a lost packet
on the end user experience. The client will only process packets in order. Therefore, a lost packet will cause
the client to buffer subsequent packets from the server and wait for the lost packet to arrive. If the resend
timeout is long, the user will notice a temporary halting of updates from the server.
▪ noSuchPortTimeout (Float)
Number of seconds of grace given to a client connection encountering a REASON_NO_SUCH_PORT error
before the connection is terminated.
The REASON_NO_SUCH_PORT error indicates that the host is no longer reachable, a condition that is
sometimes temporary, caused by intermittent network problems.
Note that this setting is only approximate ‐ the timeout is shortened if the server sends more than one
packet per tick to the client.
▪ oldStyleBaseDestroy (Boolean)
This tag is deprecated, and its use is not recommended. This option is planned for removal in BigWorld 2.0.
Flag indicating whether the destruction of a base entity should be carried out like in BigWorld versions
earlier than 1.6.

BaseApp Configuration Options
In those versions, the base entity could be destroyed before the cell entity; doing so would also destroy
the cell entity. From BigWorld 1.6 and later versions, the base entity must be destroyed after the cell entity.
Calling the method Base.destroy on an entity that still has its cell part raises a Python exception.
If this option is set to true, calling the method Base.destroy on an entity with a cell part causes the method
Base.destroyCellEntity to be called. After the cell entity has been destroyed, it the callback onLoseCell is
not implemented on the base entity, then the base entity is destroyed.
▪ pythonPort (Integer)
Port that the Python telnet session listener should listen on.
If set to zero, then a random port is chosen.
If the option is missing, then the port number will be set according to the formula:
40,000 + BaseApp ID
If the desired port is not available in any case, then a random one is used.
This option is also available for CellApps.
▪ reservedTickFraction (Float)
Percentage of tick time that should be remaining on current tick so the next one is considered to be pending.
This value is expressed as fraction. For example, setting it to 0.2 means that the next tick will be considered
pending when there is still 20% of the current tick's time remaining.
Increasing this parameter will make the server more conservative in its CPU usage.
This affects how aggressive the method BigWorld.fetchFromChunks will be about yielding processing
to the next tick.
Note: This should rarely be changed from the default value.
▪ sendAuthToClient (Boolean)
Flag indicating whether BaseApps must send authentication messages to clients (clients always send
authentication to the server).
Use this feature to avoid hacking of clients, and prevent users from spoofing server messages to other
clients.
Without this authentication, someone can send fake messages to clients, pretending to be the server (they
will need the IP address of the client, and the port that the server is using, which can only be determined
from the target client's data stream).
This option's value defaults to false, in order to save bandwidth.
▪ sendWindowCallbackThreshold (Float)
The fraction of an entity channel's send window that needs to be used before the onWindowOverflow
callback is called on the associated Base entity.
▪ shouldResolveMailBoxes (Boolean)
Flag indicating whether a mailbox shoud be resolved to a Base entity, when possible. If a mailbox refers to
a Base entity on the local BaseApp, the entity is used instead of the mailbox.

BaseApp Configuration Options
Although it is more efficient to have this option set, it is generally better to have it disabled. Having this
enabled can lead to hard to find errors as behaviour changes depending on whether an entity happens
to be local or not.
▪ verboseExternalNub (Boolean)
Flag indicating whether to generate verbose log output related to external network traffic.
▪ warnOnNoDef (Boolean)
Flag indicating whether to generate a warning when Base entity properties are set that do not have a
description in the entity's def file.
▪ watcherValues (String)
This is not an actual configuration option, but instead a sub-section inside the section , used
to set arbitrary watcher values for the BaseApp at initialisation time.
This might be useful when there is a watcher value with no corresponding entry on file /server/
bw.xml.
For example, to set value of debug/baseapp.cpp to 2:

2
...

Do not use this feature if there is a parameter that can be set directly. Like all configuration options,
this one is only evaluated once. It means that if there is an entry for watcherValues in /server/
defaults.xml, then all tags defined in it will be ignored if /server/ bw.xml also has an entry
for watcherValues (even if different tags are specified in each one). This option is also available for
CellApps.

File hierarchy

▪ writePythonLog (Boolean)
Flag indicating whether to write the output
/python.log.

Python

scripts

the

local

log

file

This should generally be set to false, since the data is sent to the central logger anyway, and the disk I/O
generated by logging could block the process.

Secondary Database Configuration Options
This option is also available for CellApps.

2.7.1. Secondary Database Configuration Options
The BaseApp secondary database configuration options are specified in file /server/ bw.xml under
section /, and are described below:
▪ enable (Boolean)
Flag indicating whether to use secondary databases.
For more details about secondary databases, see Server Programming Guide 's section Secondary
Databases.
▪ maxCommitPeriod (Float)
Maximum number of seconds between each commit. Higher values will result in better performance.
Lower values will reduce the amount of data loss in case of a total system crash.
Explicit BigWorld.writeToDB calls will always result in a commit. This option only affects the automatic
archiving of entities.
If set to zero or this option is empty, the value defaults to 5.
▪ directory (String)
Directory where the secondary database files are stored. Secondary databases are SQLite files.
The secondary database files are cleaned up when the system shuts down. However, these files should
not be treated as normal temporary files since they are crucial to the data recovery process in case of a
complete system crash.
If the first character of the path is a / character, the path is treated as an absolute path. Otherwise, the path
is treated as relative to the first path that contains the directory.

2.7.2. Packet Log Configuration Options
The BaseApp packet logging configuration options are specified in file /server/ bw.xml under
section /, and are described below:
▪ addr (String)
Client for which the packet's content should be logged.
The address is specified as dotted decimal format (e.g., 10.40.1.1).
If this option is empty, then all packets will be logged.
▪ enable (Boolean)
Flag indicating whether to write all packets to a local log file proxy.log.
This can be useful for debugging.
▪ flushMode (Boolean)
Flag indicating whether the log file should be flushed after each write.
This option is useful to ensure all log writes are captured if the BaseApp is crashing.
▪ hexMode (Boolean)

ID Configuration Options
Flag indicating whether the logged packets' contents should be written in hexadecimal.

2.7.3. ID Configuration Options
The BaseApp ID configuration options are specified in file /server/bw.xml under section
/, and are described below:
▪ criticallyLowSize (Integer)
Minimum number of IDs in the BaseApp's available ID pool before the other limits are automatically
adjusted.
The adjustment aims to help avoid this from occurring again.
▪ desiredSize (Integer)
Target number of IDs in the BaseApp's available ID pool when requesting IDs to the parent broker ID (in
case it fell below lowSize), or returning IDs to it (in case it rose above highLevel) ‐ for CellApps and
BaseApps, the parent ID broker is the CellAppMgr, while for CellAppMgr it is DBMgr.
▪ highSize (Integer)
Maximum number of IDs in the BaseApp's available ID pool before IDs are returned to the parent ID broker
‐ for CellApps and BaseApps, the parent ID broker is the CellAppMgr, while for CellAppMgr it is DBMgr.
ID recycling is currently disabled, so this value is actually never used.
▪ lowSize (Integer)
Minimum number of IDs that should be available in the BaseApp's available ID pool before a request is
sent to the parent ID broker to restore it to the value specified in configuration option desiredSize ‐ for
CellApps and BaseApps, the parent ID broker is the CellAppMgr, while for CellAppMgr it is DBMgr.

2.7.4. Client Upstream Limits
The client upstream bandwidth can be limited so as to prevent denial-of-service attacks from malicious
clients, or errant script code that cause messages from the client to be sent in high volume. This can lead to
Mercury channels within the server to become heavily loaded.
To prevent this, limits can be specified on the count and size of incoming messages from clients to BaseApps.
Once hard limits are reached on the count/size of incoming messages, messages are buffered and played
back over time. Once the hard limits on buffering are reached, clients are disconnected.
The configuration parameters are specified in the file /server/bw.xml in the /
section. The available sub-elements are described in the following table.
▪ warnMessagesPerSecond (Integer)
This is a warning limit for the number of received messages from a client that are dispatched, measured
in number of messages per second. When messages are received above this limit, a warning is emitted.
No further warnings for this client of this type are emitted until the incoming message frequency drops
below this limit.
▪ maxMessagesPerSecond (Integer)
This is a hard limit for the number of received messages from a client that are dispatched, measured
in number of messages per second. When messages are received above this limit, those and subsequent
messages are buffered for later playback until the buffer is empty. The message queue for all clients are
checked every tick to replay any eligible messages.

BaseAppMgr Configuration Options
▪ warnBytesPerSecond (Integer)
This is a warning limit for the throughput (in bytes) of received messages that are dispatched, measured
in bytes per second. When this limit is exceeded, a warning is emitted. No further warnings for this client
of this type are emitted until the incomng message data throughput drops below this limit.
▪ maxBytesPerSecond (Integer)
This is a hard limit for the size of data (in bytes) in received messages that are dispatched per second. When
messages are received above this limit, those and subsequent messages are buffered for later playback
until the buffer is empty. The message queue for all clients are checked every tick to replay any eligible
messages.
▪ warnMessagesBuffered (Integer)
This is a warning limit for the number of received messages that may be buffered from the client. When
this limit is exceeded, a warning is emitted. No further warnings for this client of this type are emitted
until the number of queued messages drops below this threshold as a result of queue playback.
▪ maxMessagesBuffered (Integer)
This is a maximum limit for the number of received messages that may be buffered from the client. When
this limit is exceeded, the client is disconnected and any messages that are buffered are discarded and not
dispatched.
▪ warnBytesBuffered (Integer)
This is a warning limit for the total number of bytes that may be buffered for a client. When this limit is
exceeded, a warning is emitted. No further warnings for this client of this type are emitted until the total
size of buffered messages drops below this threshold as a result of queue playback.
▪ maxBytesBuffered (Integer)
This is a maximum limit for the total number of bytes that may be buffered for a client. When this limit is
exceeded, the client is disconnected and any messages that are buffered are discarded and not dispatched.

2.8. BaseAppMgr Configuration Options
The BaseAppMgr configuration options are specified in file /server/bw.xml under section
baseAppMgr, and are described below:
▪ baseAppOverloadLevel (Float)
Minimum load level that all BaseApps should reach for the system to be considered in overload, and thus
reject new logins.
Similar overload levels are specified for for any CellApp (by option ‐ for
more details, see “CellAppMgr Configuration Options” on page 32), and for DBMgr (by option
‐ for more details, see “DBMgr Configuration Options” on page 35 ).
▪ baseAppTimeout (Float)
Number of seconds for a BaseApp to respond before it is considered dead.
▪ hardKillDeadBaseApps (Boolean)
Determines if a non-responsive BaseApp will be terminated with a SIGQUIT signal.
Non-responsive BaseApps must be terminated in order for its backup to take over its IP address and
ID. BaseApp non-responsiveness is determined by its backup, so a BaseApp running without a backup

Bots Configuration Options
will never be reported as being non-responsive (see also configuration option baseApp/backupTimeout in
“BaseApp Configuration Options” on page 15 ).
When this option is set to false, no signal is sent to the non-responsive BaseApp.
Only use this option for debugging, e.g., to attach a debugger to the hung process.
▪ internalInterface
For details, see “General Configuration Options” on page 7 .
This tag is deprecated, and its use is not recommended. For details, see the document Server Overview's section
Server Components → BWMachined → BWMachined interface discovery.
▪ internalLatencyMax
For details, see “General Configuration Options” on page 7 .
▪ internalLatencyMin
For details, see “General Configuration Options” on page 7 .
▪ internalLossRatio
For details, see “General Configuration Options” on page 7 .
▪ overloadLogins (Integer)
Maximum amount of logins that will be accepted during the overload tolerance period (see the
overloadTolerancePeriod option) before rejecting any further logins.
▪ overloadTolerancePeriod (Float)
Number of seconds that logins will be accepted during a situation where the BaseApps are overloaded
(see the baseAppOverloadLevel option). After this period of time, any further logins will be rejected.
▪ timeSyncPeriod (Float)
Number of seconds between each synchronisation of game time between BaseAppMgr and CellAppMgr.
▪ useNewStyleBackup (Boolean)
Old-style backup is deprecated and its use is not recommended. It is planned for removal in BigWorld 2.0.
Flag indicating whether Distributed BaseApp Backup method should be used, instead of the older Nondistributed BaseApp Backup method.
In the Distributed BaseApp Backup method, a BaseApp's entities are backed up in various other real
BaseApps, while in the Non-distributed BaseApp Backup method, various BaseApps would back up all
their entities on a dedicated Backup BaseApp.
For details, see “BaseApp Fault Tolerance” on page 86, and the document Server Overview's section
Server Components → BaseApp → Fault tolerance.

2.9. Bots Configuration Options
The Bots configuration options are specified in file /server/bw.xml under the section , and
are described below:
▪ controllerData (String)

Bots Configuration Options
Default data that the bot's controller will be created with (when bots are created, they get a controller
associated with them to control their movement).
This may have different meanings for different controller types. For example, some controller types may
interpret this as a filename to load data from.
▪ controllerType (String)
Type of the controller to be created with bot.
▪ password (String)
Password that the bots should use when logging in to the server.
▪ port (Integer)
Port on the server machine in which bots process will log to (only used if the option is
specified).
Ignored if bots automatically locates the LoginApp.
▪ pythonPort (Integer)
Port that the Python telnet session listener should listen to.
▪ randomName (Boolean)
Flag indicating whether a random suffix should be added to each bots name, in order to make them unique.
You should only set this option to false if you plan to use only a single bot. If you plan to use more than one
bot, then you will need them to have different names ‐ otherwise only the first one will be able to log in.
See also option username.
▪ scripts (Boolean)
Flag indicating whether the bots process should use Python scripting for received entities.
If set to false, then received entities are effectively ignored. Turning this option on has a significant
performance penalty.
▪ serverName (String)
Name of the server machine that the bots process should log in to (i.e., the machine running LoginApp).
If this option is empty, then the bots process will attempt to find an appropriate LoginApp on the local
network.
▪ shouldLog (Boolean)
Flag indicating whether messages generated by the bots process should be sent to the central logger.
▪ standinEntity(String)
Default entity script to be used when a specific Entity type does not have its corresponding game script.
▪ loginMD5Digest(String)
MD5 digest string (in hex readable form) for server login.
▪ userName (String)

CellApp Configuration Options
Username that bots should use when logging in to the server.
When randomName is true, this is the prefix before the randomly generated part of the name.
See also option randomName.

2.10. CellApp Configuration Options
The CellApp configuration options are specified in file /server/bw.xml under the section
, and are described below:
▪ backupPeriod (Float)
Number of seconds between backups of each cell entity to its base entity. This value is rounded to the
nearest game tick.
As a first level of fault tolerance, cell entities are copied to their base entities, while base entities can
be copied to their backup BaseApps. For more details on BaseApp and CellApp fault tolerance, see the
document Server Programming Guide's chapter Fault Tolerance.
The value for this option is very dependant on the game. A small value means frequent backups, and
consequently less lost data in case a CellApp fails. But backups cost bandwidth and CPU on the CellApp.
See also options archivePeriod and backupPeriod on “BaseApp Configuration Options” on page
15 .
▪ checkOffloadsPeriod (Float)
Number of seconds between offload checks.
This is a periodic check if entities need to be offloaded, or new ghosts created.
Other options affected by this setting are: ghostingMaxPerCheck and offloadMaxPerCheck.
▪ chunkLoadingPeriod (Float)
Number of seconds between checks on the progress of loading and unloading chunks.
Chunk loading occurs in a separate thread, but this check in the main thread queues up more chunks for
the loading thread to load (or unload).
▪ defaultAoIRadius (Float)
The default AoI radius of new cell entities for proxy entities when they are created. See also the CellApp
Python API documentation's entry Main → Cell → BigWorld → Classes → Entity → setAoIRadius.
Note: If this is larger than GHOST_DISTANCE (500m), then the AoI radius of player entities is clamped to
GHOST_DISTANCE across cell boundaries.
▪ enforceGhostDecorators (Boolean)
Specifies whether to enforce the requirement of adding a decorator to methods that can safely be called
on ghost entities.
When enabled, methods that have not been labelled as safe and are called on an entity that could be a ghost
will generate a Python exception. To be considered safe, the method must either be described in the .def
file or be decorated with @bwdecorators.callableOnGhost.
import bwdecorators

CellApp Configuration Options

class Table( BigWorld.Entity ):
@bwdecorators.callableOnGhost
def getArea( self ):
return self.width * self.height

▪ entitySpamSize (Integer)
Number of bytes that an entity can send to another before a warning message is displayed.
This can be useful to identify entities that are causing a lot of network traffic.
▪ fastShutdown (Boolean)
Specifies whether to avoid normal chunk unloading when the system is being shut down. This
considerably speeds up the shutdown process.
▪ ghostingMaxPerCheck (Integer)
Maximum number of ghost creation messages that can be sent per offload check.
The frequency of the offload check is determined by option checkOffloadsPeriod.
▪ ghostUpdateHertz (Integer)
Number of seconds between the flushing messages to channels of neighbouring CellApps.
Channels are created between neighbouring CellApps. Messages (such as ghost data) sent over these
channels are not sent immediately, but are instead flushed periodically. This is done to avoid the high
overhead of sending a packet.
If the value of this option is decreased, then there will be more lag for cross-cell communications.
Be careful about increasing this value, because CellApps have a fixed window size, after which they are
flagged as dead. This window is currently 4096, which mean that if 4,096 packets are not acknowledged,
then the CellApp is flagged as dead. At 50Hz, this takes around a minute, which is enough to survive
occasional heavy bouts of unexpected swapping.
Note: Bases always flush messages immediately.
▪ internalInterface
For details, see “General Configuration Options” on page 7 .
This tag is deprecated, and its use is not recommended. For details, see the document Server Overview's section
Server Components, BWMachined, BWMachined interface discovery.
▪ internalLatencyMax
For details, see “General Configuration Options” on page 7 .
▪ internalLatencyMin
For details, see “General Configuration Options” on page 7 .
▪ internalLossRatio
For details, see “General Configuration Options” on page 7 .
▪ loadDominantTextureMaps (Boolean)
Specifies whether to load the terrain's dominant texture maps. By default, this flag is set to false.

CellApp Configuration Options
Loading the dominant texture maps enables using features such as material kinds returned by
BigWorld.collide.
▪ loadSmoothingBias (Float)
Value to smooth the load changes on a component by when calculating new load.
The BigWorld server uses the load on a component to perform its load balancing. Unfiltered, the load can
change too quickly to be useful. The option loadSmoothingBias is used to smooth out this value.
The filtered value is calculated at each game tick as follows:
newSmoothedLoad = (1 - loadSmoothingBias) * oldSmoothedLoad + loadSmoothingBias
* load
This option is also available for CellAppMgr and BaseApps.
▪ maxControllersAbsolute (Integer)
Number of controllers that an entity must have before an exception is raised on attempts to create a new
controller on it.
▪ maxControllersExpected (Integer)
Minimum number of controllers an entity must have before a warning is generated on attempts to create
a new controller on it.
▪ maxGhostsToDelete (Integer)
Maximum number of ghosts to be deleted from other cells on every offload check (the frequency of this
check is set via checkOffloadsPeriod).
This option is useful for adding antihysteresis and for smoothing the load caused by ghost deletion.
▪ minGhostLifespan (Float)
Minimum number of seconds for which a real entity will keep a ghost one.
This is useful for adding antihysteresis to the ghost creation and to the deletion process.
▪ maxPhysicsNetworkJitter (Float)
Maximum number of seconds to allow for when network jitter when considering movement cheating.
The movement of a player may vary slightly due to variations in network latency. This value sets the level
of tolerance for this jitter.
▪ navigatorUseGirthGrids (Boolean)
Specifies if the waypoint search optimisation scheme should be used.
Girth grid is an optimisation scheme for waypoint search in a chunk. When this scheme is used, a chunk
is divided up into a set of 12x12 grids according to the girth provided (you can have a list of 12x12 grid
set for different girth sizes). Every grid square contains a subset of waypoints that overlap the covered
area in a chunk.
During a (waypoint) search, only targeted grid squares (i.e., subset of waypoints) are searched for the
waypoint, instead of searching through the full set of waypoints. This scheme will generally improve the
waypoint search performance.

CellApp Configuration Options
▪ obstacleTreeDepth (Integer)
Depth of the obstacle tree to create.
Higher numbers increase the speed of collision detection but use more memory.
▪ offloadHysteresis (Float)
Number of metres that an entity has to be over a cell boundary before it is actually offloaded.
This helps avoid entities being offloaded back and forth between cells when standing very close to the
border.
▪ offloadMaxPerCheck (Integer)
Maximum number of entities that can be offloaded from a cell during an offload check.
The frequency of the offload check is determined by option checkOffloadsPeriod.
▪ pythonPort (Integer)
Port that the Python telnet session listener should listen on.
If set to zero, then a random port is chosen.
If the option is missing, then the port number will be set according to the formula:
50,000 + CellApp ID
If the desired port is not available in any case, then a random one is used.
This option is also available for CellApps.
▪ reservedTickFraction (Float)
Percentage of tick time that should be remaining on current tick so the next one is considered to be pending.
This value is expressed as fraction. For example, setting it to 0.2 means that the next tick will be considered
pending when there is still 20% of the current tick's time remaining.
Increasing this parameter will make the server more conservative in its CPU usage.
This affects how aggressive the method BigWorld.fetchFromChunks will be about yielding processing
to the next tick.
Note: This should rarely be changed from the default value.
▪ sendWindowCallbackThreshold (Float)
The fraction of an entity channel's send window that needs to be used before the onWindowOverflow
callback is called on the associated entity.
▪ shouldResolveMailBoxes (Boolean)
Flag indicating whether a mailbox shoud be resolved to a Cell entity, when possible. If a mailbox refers to
a Cell entity on the local CellApp, the entity is used instead of the mailbox.
Although it is more efficient to have this option set, it is generally better to have it disabled. Having this
enabled can lead to hard to find errors as behaviour changes depending on whether an entity happens
to be local or not.
28

CellApp Configuration Options
▪ treatAllOtherEntitiesAsGhosts (Boolean)
Puts the CellApp in a debugging mode in which a script running on it will see only its own entity as real
‐ all other entities will be treated as ghosts.
Method calls, property access, and other functions will operate as if the other entities really are ghosts.
This mean that:
▪ Method calls will go via the network.
▪ Property access will be read-only, and limited to CELL_PUBLIC (or more public) properties (for more
details, see the document Server Programming Guide's section Properties → Data distribution).
▪ Many internal functions will not work, e.g., adding a new Controller.
This is very useful for testing how your scripts work when dealing with ghost entities, especially if two
interacting entities are nearby and would consequently rarely be ghosts.
▪ watcherValues (String)
This is not an actual configuration option, but instead a sub-section inside the section cellApp, used to
set arbitrary watcher values for the CellApp at initialisation time.
This might be useful when there is a watcher value with no corresponding entry on /server/
bw.xml.
For example, to set the watcher value debug/cellapp.cpp to 2:

2
...

Do not use this feature if there is a parameter that can be set directly.
Like all configuration options, this one is only evaluated once. It means that if there is an entry for
watcherValues in /server/defaults.xml, then all tags defined in it will be ignored if /
server/bw.xml also has an entry for watcherValues (even if different tags are specified in each one).
This option is also available for BaseApps.

File hierarchy

▪ writePythonLog (Boolean)
Indicates if output of Python scripts should be written to /python.log.

Noise Configuration Options
This should generally be set to false, since the data is sent to the central logger anyway, and the disk I/O
generated by logging could block the process.
This option is also available for BaseApps.

2.10.1. Noise Configuration Options
The CellApp noise configuration options are specified in file /server/bw.xml under section
/, and are described below:
▪ horizontalSpeed (Float)
If an entity's horizontal speed exceeds this value (in metres per second), the entity makes a noise.
See Entity.makeNoise script method for more information. The event and info are 0 for noises
generated this way.
▪ standardRange (Float)
Distance in metres through which a noise is propagated.
This value is multiplied by the level of a noise. For details, see the CellApp Python API documentation's
entry Main → Cell → BigWorld → Classes → Entity → makeNoise.
▪ verticalSpeed (Float)
If an entity's falling speed exceeds this value (in metres per second), the entity makes a noise.
This is done via script method Entity.makeNoise ‐ for noises generated this way, the parameter event
and info are set to 0. For details, see the CellApp Python API documentation's entry Main → Cell →
BigWorld → Classes → Entity → makeNoise.

2.10.2. ID Configuration Options
The CellApp ID configuration options are specified in file /server/bw.xml under section
/, and are described below:
▪ criticallyLowSize (Integer)
Minimum number of IDs in the CellApp's available ID pool before the other limits are automatically
adjusted.
The adjustment aims to help avoid this from occurring again.
▪ desiredSize (Integer)
Target number of IDs in the CellApp's available ID pool when requesting IDs to the parent broker ID (in
case it fell below lowSize), or returning IDs to it (in case it rose above highSize) ‐ for CellApps and
BaseApps, the parent ID broker is the CellAppMgr, and for CellAppMgr it is DBMgr.
▪ highSize (Integer)
Maximum number of IDs in the CellApp's available ID pool before IDs are returned to the parent ID broker
‐ for CellApps and BaseApps, the parent ID broker is the CellAppMgr, and for CellAppMgr it is DBMgr.
ID recycling is currently disabled, so this value is actually never used.
▪ lowSize (Integer)

CellApp Profiles Configuration Options
Minimum number of IDs that should be available in the CellApp's available ID pool before a request is
sent to the parent ID broker to restore it to the value specified in configuration option desiredSize ‐ for
CellApps and BaseApps, the parent ID broker is the CellAppMgr, and for CellAppMgr it is DBMgr.

2.10.3. CellApp Profiles Configuration Options
The CellApp profiles configuration options are specified in file /server/bw.xml under section
/. It contains sub-sections for enabling profiling of specific CellApp functionality.
All the profiling options specified below can be modified after server startup on a per CellApp basis by using
Watchers. The values are exposed in the CellApp watcher tree under profilesConfigs/

Example file /server/bw.xml

The file name might have the suffix random, in which case the start position of the bot is chosen randomly
from the set of node positions in the graph.
The format specification for the movement graph file is illustrated below:

104

NodeProperties Section

?NodeProperties

integer integer integer
? string_node_name

?NodeProperties
*
string_node_name

Grammar of movement graph files

The list below describes the tags in the movement graph file:
▪ edge (section nodes/node/edges)
Name of one of the nodes that this node borders. Must be the same value specified for a name section
in the file.
▪ edges (section nodes/node)
Tag specifying all the nodes that this node borders.
▪ name (section nodes/node)
Name used to refer to the node. If this tag is not specified for a node, then you can refer to it by an integer
sequential number, with the first node having an index of zero.
▪ node (section nodes)
Tag for section specifying a specific node in the graph.
▪ nodeDefaults
Tag for section specifying default values for all nodes. These values can be overridden on a per-node basis
in section nodes/node.
▪ NodeProperties (section NodeDefaults and nodes/node)
For details, see “NodeProperties Section” on page 105 .
▪ nodes
Tag for section specifying all the nodes in the graph.
▪ pos (section nodes/node)
XYZ position of the centre of the node. The node is the area around pos, extending for n metres around
it (n being the value of NodeProperties' tag radius).

8.4.1. NodeProperties Section
The format specification for the NodeProperties section is illustrated below:
?

float

105

Extending Bots

?
?
?
?

float
float
float
float

Grammar of NodeProperties section

The list below describes the tags in the NodeProperties section:
▪ maxSpeed
Maximum speed of the bot.
▪ maxStay
Maximum number of seconds that the bot should stay in the node.
▪ minSpeed
Minimum speed of the bot.
▪ minStay
Minimum number of seconds that the bot should stay in the node.
▪ radius
Number of metres around the node's pos location to be considered as the node. When a bot is travelling
between two nodes, it chooses a random point of arrival in the destination node. This way, the bots do not
follow the exact same line. The point is chosen to lie within the radius of the destination node.
An example movement graph file is displayed below:

5
10

Town1

5000 0 0

Town2

5000 0 5000

Example movement graph file

8.5. Extending Bots
8.5.1. Creating New Movement Controllers
The default movement controller is Patrol, but you might want to create one that better represents the
expected traffic patterns and entity distributions of your game, or to stress test other aspects of the system.

106

Extending Bots
To create a new movement Controller, derive its class from the MovementController class. Also derive a class
from the MovementFactory class. The factory is used to parse the Controller data argument, and instantiate
a new movement controller, while the movement controller moves the bot around.
The fragments below list the relevant methods in the base classes:
▪ Class MovementController
bool nextStep( float speed, float dTime, Vector3& position, Direction3D&
direction)

▪ Class MovementFactory
MovementFactory(const char* name)
MovementController* create(
const std::string& data,
const Vector3&
startPosition)

107

Chapter 9. Security
Security has been of paramount importance in the design and implementation of all parts of BigWorld.
The basic philosophy is to always handle with care any client-initiated actions or messages. This should be
accomplished in a way that does not unduly limit potential game designs.
For more details, see the document Server Programming Guide, chapter Security Systems in BigWorld
Technology.

9.1. Security of the Server
The internal network is assumed to be secure — BigWorld does not implement security measures to
safeguard processes against an attacker gaining access to the cluster's internal LAN. Operators should
ensure that the usual protections for an internal network are in place. Remote access should be very strictly
controlled.
The external points of contact are the area of most concern when running an exposed server. For BigWorld,
these are LoginApp and BaseApp, as illustrated below:

BigWorld Server components

Due to the bandwidth needs of a massively multiplayer online game, LoginApp and BaseApp are intended to
be run on machines with external access. In some sense they are the firewall. For more details, see “Blocking
Ports and Related Security Considerations” on page 110 .

108

Server Ports
LoginApp receives only fixed-length queries, making it easy and transparent to secure. This process is
expected to be tailored by customers to suit their game, but care should be taken when doing so.
BaseApp receives more complex data, including script method calls, and is the gateway to the rest of the
BigWorld Server. It has many checks in place to ensure the integrity of received data, and to discard (and
warn about) corrupted data and hacking attempts. The string CHEAT is used in the log messages when
BigWorld receives potentially malicious data that does not conform to its protocols (the CellApp may also
use this indicator.) It is advised that MessageLogger logs be monitored for messages containing that string
(for details on MessageLogger, see “MessageLogger” on page 50 ).
The security of the game-level logic rests to a certain extent with the Python scripts that implement it. For
example, an entity should not be able to stab another entity that is 100 metres away. For more details on this
topic and on server features such as physics checking, see the document Server Programming Guide.

9.2. Server Ports
The list below describes the ports used by BigWorld server:
▪ 20013 (Protocol: UDP, Access: External)
Used by LoginApp, this port can be overridden in the following ways:
▪ In /server/bw.xml file's loginApp section, set the port configuration option.
▪ In the command-line arguments to LoginApp, use the -loginPort option.
▪ 20017 (Protocol: UDP, Access: Internal)
The HTTP interface of Watcher. To start the interface, run bigworld/tools/server/watcher, then
connect to this port with a web browser.
▪ 20018 (Protocol: UDP, Access: Internal)
Used by BWMachined.
▪ 40001-49999 (Protocol: TCP, Access: Internal)
The Python server on BaseApp.
All BaseApps have a Python server that can be telnetted to. The port number is 40000, plus the BaseApp ID.
BaseApp ID numbers start at 1, so to talk to the third started BaseApp, telnet to 40003.
▪ 50001-59999 (Protocol: TCP, Access: Internal)
The Python server on CellApp.
All CellApps have a Python server that can be telnetted to. The port number is 50000, plus the CellApp
ID. CellApp ID number start at 1, so to talk to the third started CellApp, telnet to 50003.

Note
CellApp ID is not the same as the Cell ID. Cell IDs are allocated sequentially, as they
are allocated to a cluster, not when the CellApp is started.
▪ 32768-61000 (Protocol: TCP, Access: Internal)
Used by CellAppMgr and CellApp for viewer applications such as Space Viewer. Automatically assigned
by the kernel (see UDP entry).

109

Blocking Ports and Related Security Considerations
▪ 32768-61000 (Protocol: UDP, Access: Internal [External for BaseApp])
Used by the server components: CellApp, CellAppMgr, BaseApp, BaseAppMgr, DBMgr, MessageLogger
and StatLogger. Only the BaseApp has an external port.
Automatically assigned by the kernel, in the range of the kernel setting /proc/sys/net/ipv4/
ip_local_port_range, which defaults to 32768-61000.
The BaseApp external port may be exempted from random assignment, by specifying it in the /
server/bw.xml file's baseApp section's externalPort configuration option.
The port chosen can be displayed via the Watcher interface under:
▪ /(internal|external) nub/address on BaseApp.
▪ /nub/address on other components.

9.3. Blocking Ports and Related Security Considerations
Since TCP/IP is not used externally, you can block all TCP traffic. Leave all UDP/IP ports 32768 and above
open, as well as the login port (20013 by default).
Use of a separate firewall machine is strongly discouraged. The BaseApp machines are designed to be the
firewalls themselves, and perform a very similar proxying function for clients. Their amount of processing
is small enough so that they can handle a whole network adapter's worth of Internet traffic. Adding another
machine would only be a waste of hardware, maintenance time, and latency. BaseApps only listen on one
UDP port — and so the whole TCP stack can be disabled on the external interface. The use of standard
firewall software such as iptables may be an appropriate way to accomplish this.
Network tools such as lsof and netstat should be consulted, to ensure that you are not running any daemons
listening on the external (or common) interface. Apart from BWMachined, you should not need to run
any daemons with BigWorld, but if you wish to, then you should ensure that their ports are blocked.
BWMachined should not listen on the external interface, only the internal interface.
Barring all TCP packets greatly improves the security of a BaseApp machine. TCP is a complicated protocol,
and requires many tables and buffers to implement or firewall. By this rationale, the security of a BaseApp
machine may be considered even better than that of an ordinary firewall, which must conditionally pass
TCP packets.
To reduce exposure to DDOS attacks, it is recommended that the BaseApp be left to choose, within the range
allocated by the operating system, a random port. This way, if an attacker discovers the IP and port of one of
the BaseApp machines, then it does not mean he will automatically know those details for the other BaseApp
machines.

110

Chapter 10. BigWorld Server Across Multiple
Machines
To start a server cluster consisting of multiple machines, you can start each process in any of the following
ways:
▪ the command line
▪ the control_cluster.py script
▪ WebConsole's ClusterControl module
CellApps, BaseApps, and LoginApps can have multiple instances running, while the other processes can
have only one. The clients should connect to the IP address of a machine that is running a LoginApp.

10.1. How To Start
The following sub-sections describe how to start server components.

10.1.1. WebConsole
WebConsole can be used to easily start, stop and control server components.
For an outline of WebConsole, see “WebConsole” on page 45 . For operational behaviour, see the online
WebConsole documentation in the Cluster Control module.

10.1.2. Auto Configuration Via control_cluster.py
The script bigworld/tools/server/control_cluster.py is generally the easiest way to start a multimachine BigWorld Server. For details, see “control_cluster.py” on page 67 .

10.1.3. Hard-Coded Scripts
Create your own shell scripts using the script bigworld/tools/server/misc/startsingle.sh as a
guide.

10.1.4. Manual Start
During development and testing, it is feasible to manually start processes individually.

10.2. How To Stop
The method used to stop the system depends on the method used to start it. For example, calling the script
control_cluster.py with the option stop stops a system started with that script. Stopping LoginApp
triggers other processes to terminate, unless this feature is disabled in configuration file /server/
bw.xml.

10.3. How To Monitor
Use WebConsole (for details, see “WebConsole” on page 45 .) and control_cluster.py (for details, see
“control_cluster.py” on page 67 .) for monitoring and recording real-time statistics. You can also get
profiling data from the watchers (via WebConsole's ClusterControl module). Use WebConsole's LogViewer
module to monitor the centralised log.

10.4. LoginApp and Scalability
Just as you can use multiple CellApps and BaseApps in a large cluster, you can also run multiple LoginApps
in a load sharing arrangement.

111

LoginApp and Scalability
To do that, DNS has to set up so that it returns multiple DNS addresses for the login server's name. If you
are using BIND, then you can simply put multiple A addresses in the zone configuration file. This means
that when a client looks up the DNS address of the login server, it receives a login server randomly selected
from the available pool.
For example, if you are using BIND as your DNS server (most Linux distributions come with BIND as their
default DNS server), the zone file can have configuration similar to the ones in the table below:

Name TTL CLASSTYPE ResourceRecord
login

600 IN

10.0.0.1

600 IN

10.0.0.2

600 IN

10.0.0.3

Example configuration on a BIND zone file

Because there are multiple A records with different IP addresses under one name, the IP address returned
to a client when it looks up the address for the login server will be picked up from the IP address list, in a
round robin manner.

112

Chapter 11. Multiple BigWorld Servers in a Single
LAN
11.1. Keeping Processes Separate
All server components in a BigWorld server cluster run under the same UID. If you want to run multiple
server clusters on the same network (for different stages of testing, for example), then make sure that you
use different UIDs.
You also need to make sure that the LoginApps are on different machines. This is because by default each
LoginApp opens the same specific port, and therefore only one could succeed in binding to it. Alternatively,
multiple LoginApps can be run on a single machine as long as they have different port settings in /
server/bw.xml.

Note
It is important the username and UID mapping is consistent across all machines on the
cluster. It is highly recommended to use centralised logins using LDAP or similar.

11.2. Centralised Cluster Monitoring
The best way to keep track of processes is via WebConsole's StatGrapher module.
The module, which can be run on any web browser, uses log files generated by StatLogger daemon to display
statistics, and has a variety of options to filter the output by user.
For more details, see “StatGrapher” on page 47 , and “StatLogger” on page 55 .

11.3. Auto-Detection of LoginApps
When running multiple servers, it is convenient for the client to auto-detect LoginApp processes (this is only
suitable for a development environment).
The LoginApps have this functionality built in, and do not require any configuration to perform it — they
register themselves with BWMachined, and the client broadcasts a 'find' message to retrieve their contact
details. A probe message is then sent to each server to determine who, where and what it is running. This
functionality is implemented on the client by class ServerDiscovery.

113

Chapter 12. DBMgr MySQL Support
Following is described the minimal set of steps required to enable MySQL support for a server cluster. They
assume that a MySQL database has been installed and configured as outlined in the Server Installation Guide,
section Installing the BigWorld Server, Configure MySQL server.

12.1. Compiling DBMgr with MySQL Support
The DBMgr binary provided with the default BigWorld package does not have MySQL support built-in. This
allows users to quickly start using BigWorld without having to configure a production database to test a
simple package.
First, to enable compilation of DBMgr, make sure that the MySQL development files are installed on the
build system. To do this, as root, issue:
▪ On Fedora:
yum install mysql-devel

▪ On Debian
apt-get install libmysql++-dev

Next, the DBMgr makefile needs to be modified, to enable MySQL support. Edit bigworld/src/server/
dbmgr/Makefile and change the variable definition from USE_MYSQL=0 to USE_MYSQL=1 (USE_XML can
be left on, without any side effects).
After these alterations, it will be possible to rebuild DBMgr. From within the bigworld/src/server/
dbmgr folder, issue the command make.

12.2. Update bw.xml To Use MySQL
Once DBMgr has been compiled to communicate with a MySQL server, the game resource configuration file
/server/bw.xml needs to be updated with details on username, password and the host machine
the MySQL server is running on.
The example below illustrates FantasyDemo configuration (via fantasydemo/res/server/bw.xml):

mysql

my_mysql_server_machine
bigworld
my_pass

fantasydemo

Example fantasydemo/res/server/bw.xml

For details on these fields and other relevant configuration options for your production environment, see
Server Configuration with bw.xml on page 6 .

12.3. Initialise Database With Entity Definitions
DBMgr requires the MySQL database table structure to exactly match the current entity definitions.

114

Disabling Schema-Modifying Capability
To initialise the MySQL database with the correct structure, go to the folder containing the server binaries
(e.g., mf/bigworld/bin/Hybrid), then run DBMgr with the appropriate argument:
$ ./dbmgr --sync-tables-to-defs

When run with the --sync-tables-to-defs option, DBMgr updates the table structure and terminates,
to allow the server to be started normally.

12.4. Disabling Schema-Modifying Capability
By default, DBMgr has the ability to alter the database schema through the --sync-tables-to-defs
command-line option or the / configuration option. This feature may be
undesirable in a production environment where the data in the database is highly valuable. Accidental use of
the sync-tables-to-defs feature may result in data loss. For example, if the entity definition files were deleted,
it will cause DBMgr to remove the corresponding tables in the database.
The schema-modifying capability of the DBMgr can be removed by recompiling it after setting the
variable ENABLE_TABLE_SCHEMA_ALTERATIONS to 0 in bigworld/src/server/dbmgr/Makefile.
The resulting DBMgr executable will no longer have the ability to modify the table schema. Use of the -sync-tables-to-defs command-line option or the / configuration
option will result in a warning. The DBMgr will fail to start if the database table schema does not match the
entity definitions.
The new DBMgr executable will be able to connect to a database with just the SELECT, INSERT, UPDATE
and DELETE privileges. Please note that the original DBMgr executable required the CREATE TABLE
privilege even when sync-tables-to-defs feature is not used. This is because the original DBMgr executable
automatically creates and modifies BigWorld internal tables (tables with names starting with "bigworld").
It is likely that the original DBMgr executable with the sync-tables-to-defs feature will still be useful since
manually updating the table schema to match the entity definition is a time consuming task. A copy of the
original DBMgr executable can be kept at a location where it is not likely to be invoked in the day-to-day
operations of a production server.

12.5. Enabling Secondary Databases
Secondary databases can only be enabled when MySQL support is enabled. Please see the document Server
Programming Guide's chapter Secondary Databases for details about secondary databases.
To enable secondary database support, you must:
▪ Enable MySQL support (see above).
▪ Build the data consolidation tool by issuing the make command from within the bigworld/src/
server/tools/consolidate_dbs directory.
▪ Set the bw.xml option // to true.

115

Chapter 13. RPM
BigWorld provides a RPM implementation which allows the creation of binary RPM package for
BWMachined. The RPM implementation uses the RPM system available in Linux distributions such as
Fedora to generate RPM packages. Currently, the RPM implementation only supports building RPM
packages on i386 architecture.
This chapter provides the following information:
▪ Directory structure and files related to BigWorld RPM implementation.
▪ How to generate BigWorld binary RPM packages.
▪ Customising RPM Packages.
▪ Setting up a yum repository.
▪ Install, upgrade and uninstall using yum command.
▪ How to obtain version number of an installed package.

13.1. Directory Structures and Files
The RPM implementation is located in the bigworld/tools/server/rpm directory:
▪ The Makefile is used to generate RPM packages.
▪ The generate.py is used to facilitate the creation of RPM packages.
▪ The binary_rpms directory is where generated RPM packages are placed.
▪ The rpm directory also contains package specific directories. For example, the bwmachined directory
contains all the files specific to BWMachined.

13.2. How to Generate Binary RPM Packages
Binary RPM packages must be generated using a normal Unix user account. Binary RPM packages must not
be generated using the root user.
Before generating RPM packages, the normal user used will need to be given read and write access to the the
directory used by the RPM system to build the RPM packages. Run the following command to determine
which directory is used by the RPM system to generate RPM packages:
$ rpm --eval %_topdir

To generate RPM package for BWMachined, go to the rpm directory and run the following command:
$ make

The generated RPM package will be placed in the binary_rpms directory.
The generated RPM package will have the following file name:
bigworld-bwmachined--.i386.rpm

116

Customising RPM Packages
▪ The field is the version number of the BigWorld release that the package was generated from.
This field is based on the version number in bigworld/res/version.xml.
▪ The field identifies the specific build of this RPM package. By default, it is not used and is mapped
to patch number in bigworld/res/version.xml.

13.3. Customising RPM Packages
The most common customisation of a RPM package is the configuration file(s) installed by the package.
That is, one RPM package may include the default configuration file and another may include a customised
configuration file. These RPM packages can be distinguished by using the release field in the file name
of these packages.
For example, the configuration file included in a BWMachined RPM package is located in
bwmachined/bwmachined.conf.

rpm/

To generate a BWMachined RPM package with the default configuration file and assign it the release
number 0:
▪ Use the bwmachined.conf provided by default.
▪ Update the Release tag in rpm/bwmachined/bwmachined_template.spec to the following:
Release: 0

▪ Generate the RPM package.
The file name of the RPM package generated will be, for example, bigworld-bwmachined-1.9.1.00.i386.rpm.
To generate a BWMachined RPM package with customised configuration file and assign it the release
number 1:
▪ Update the bwmachined.conf as required.
▪ Update the Release tag to 1.
▪ Generate the RPM package.
The file name of the RPM package generated will be, for example, bigworld-bwmachined-1.9.1.01.i386.rpm.

13.4. Setting up a Yum Repository
For a large environment where RPM Packages, e.g. the BWMachined RPM Package, need to be installed on
many machines, we recommend that a yum repository to be set up to provide these RPM packages. This
allows an RPM package to be installed on any machine in the cluster by running the yum command without
the need to manually copy the RPM package to each machine in the cluster.
The yum repository created will need to be made available through a web server such as the Apache HTTP
Server. The setup required is outside the scope of this document. Please consult relevant web server
documentation on how to achieve this.
Steps to set up a yum repository:
▪ On the machine that will be hosting the yum repository, create a directory for the BigWorld RPM packages.
The location of this directory will depend on the web server chosen and the web server configuration
which is outside the scope of this document.

117

Setting up a Yum Repository
▪ Copy the RPM packages to the directory created.
▪ Install the createrepo package using the following command:
$ yum install createrepo

This is a utility that will generate a common metadata repository from a directory of RPM packages.

Note
You will need to be the root user to install a package.
▪ Run the following command to create the necessary metadata for the yum repository:
$ createrepo

where is the path to the directory created.
For example, if the directory created is /mnt/bigworld_repo, then the command to run is the following:
$ createrepo /mnt/bigworld_repo

This creates a repodata directory in the /mnt/bigworld_repo directory.
▪ The /etc/yum.conf on a machine that will access the repository created will need to be updated to
include the following setting:
[]
name=
baseurl=
enabled=1

where is replaced by the name given to the repository and is replaced by the
URL that refers to the repository.

Note
By default, the /etc/yum.conf contains a [main] entry. The entry above should
be added after the [main] entry.
When the yum repository is updated, for example, a new version of BWMachined RPM package is added,
then the repodata directory should be deleted and the createrepo command should be run again to
generate up-to-date metadata.
If a new RPM package was added to the repository but is not displayed on a machine when running a query
command such as
$ yum info bigworld-bwmachined

run the following command to delete the metadata used by yum to determine the remote availability of
packages:

118

Install, Upgrade and Uninstall using Yum Command

$ yum clean meatadata

When the yum command is run the next time, it will download the up-to-date metadata.

13.5. Install, Upgrade and Uninstall using Yum Command
This section describes how to install, upgrade and uninstall using the yum command. The BWMachined
package is used as an example.

13.5.1. Install and Upgrade using a RPM Package Directly
▪ To install a RPM Package directly without using yum repository:
$ yum --nogpgcheck install bigworld-bwmachined--.i386.rpm

where and are replaced by the actual version and release number.
This assumes that you are running the yum command in a directory containing the specified .rpm file.

Note
The RPM packages created by BigWorld RPM implementation are not signed, since
these packages are from trusted source. Therefore, during installation and upgrade,
GPG check is disabled.
▪ To upgrade a package:
$ yum --nogpgcheck upgrade bigworld-bwmachined--.i386.rpm

13.5.2. Install and Upgrade using Yum Repository
▪ To install a package from a yum repository:
$ yum --nogpgcheck install bigworld-bwmachined

▪ To upgrade a package:
$ yum --nogpgcheck upgrade bigworld-bwmachined

13.5.3. Remove an Installed Package
To remove an installed package:
$ yum remove bigworld-bwmachined

13.6. How to Obtain Version Number of an Installed Package
To obtain the version number of an installed package such as the BWMachined package, run the following
command:

119

How to Obtain Version Number of an Installed
Package
$ yum info bigworld-bwmachined

120

Chapter 14. First Aid After a Crash
If a BigWorld server component fails, then there are some steps that you should follow, to allow us to identify
and resolve the problem as quickly as possible.
The first thing you should do is determine which component crashed first. There are many ways to do this
— you can examine the logs or check the cores.

14.1. Procedure For Crash Reporting
A crash may be either intentional (an assert), or unintentional (such as segmentation fault). Intentional
crashes are often due to errors in extensions, or unexpected use of script.
When you report a crash, be sure to include in your e-mail the following information:
▪ Backtrace The full call stack for the first process that crashed (use the bt command in gdb).
▪ Logs 50 lines before, and 50 lines after the crash (this can be obtained by issuing the command mlcat.py
--around --context 50), plus the last 50 lines of output generated by the process that
crashed (this can be obtained by issuing the command mlcat.py --around --context
50 --filter pid ).
If a more detailed investigation is required by BigWorld support staff, it is likely that it will be necessary
to provide us with the actual raw log data generated at the time of the crash. The interactive segment
selection mode of mltar.py (for details, see MessageLogger, “Command-Line Utilities” on page 52 ) is
useful for archiving the necessary files before sending them.
It is not necessary to send any core files immediately, but do keep them so we can examine them later.
It is also very helpful if you give us access via ssh to a machine where we can examine the core dump, thus
avoiding its time-consuming transmission over the Internet. This is also a good procedure because it allows
us to verify that the correct executable binary is being used, which is an especially important check if you
modified and relinked the executable, or have custom extensions, etc...

14.2. Q & A
Q: What if there is no core dump?
A: Some shells by default prevent the creation of core dumps. To change this default, before running
BigWorld processes issue the command 'ulimit -c unlimited'. This command sets resources limits
— in this case it sets the core files to unlimited.
Q: How to determine the first process that crashed?
A: This is the information that could be used to determine that:
▪ Date on the core dump.
▪ Log message about the unexpected death of a process.
▪ 'restore' messages in the backtrace.

121

Chapter 15. Common Log Messages
This section provides the description of common log messages.

15.1. Warnings
▪
CellApp
WARNING WatcherDoc::initWatcherDoc: unable to load
watcherdoc/cellapp.xml

This WARNING is due to an incomplete feature to add documentation to our watchers. We apologise if
it caused any confusion. You can silence this warning by creating cellapp.xml in bigworld/res/
watcherdoc. The file should contain an empty root section:

▪
CellApp
WARNING InterfaceElement::expandLength: Received a message
longer than normal length

We try to optimise network traffic by using the smallest integer size that can encode the expected length
of the message. For example, if we expect the message to be always less than 255 bytes, we will use just
one byte to encode the length. We use 2 bytes for messages that are expected to be less than 65535 bytes
long, etc. However, if the message exceeds the expected size, we will fallback to using 4 bytes to encode
the length. The above WARNING means that a message has exceeded the expected message length.
Unfortunately, the WARNING does not tell us which message has exceeded the message length. We have
added additional information to the WARNING in BigWorld 1.8.2.
▪
CellApp
WARNING
0.21 seconds

CellApp::handleGameTickTimeSlice: Last game tick took

This WARNING means that the game tick took longer than the normal 0.1 seconds. This can be due to many
reasons but usually it is due to a script function taking a long time. You should ensure that your script
code does not do anything that takes significant amount of time. You may have to break your processing
into several runs by using a timer.
▪
CellApp

WARNING

CellApp: Scaling back!!

In response to a game tick exceeding 0.1 seconds, the CellApp will skip some of its usual processing to help
decrease the load, e.g. it will reduce the amount of updates sent to the client. This WARNING is to inform
you that clients may receive less updates e.g. entity movements may be jerky.
▪
CellApp

WARNING

cellapp.cpp:1918: Profile CALL_TIMERS took 0.12 seconds

This means that the total time of all onTimer() callbacks in that tick took 0.12 seconds. This is bad because
one tick should be less than 0.1 seconds.
CellApp

WARNING

cellapp.cpp:1572: Profile GAME_TICK took 0.12 seconds

122

Errors
This means the game tick took more than 0.1 seconds. This time includes the CALL_TIMERS time above
so almost all of the game tick was in onTimer() callbacks.
▪
CellApp
packets)

WARNING

Witness::update: 474 has a deficit of 2370 bytes (10.68

This is related to the configuration in bw.xml. If
is 20000, then it means that the maximum number of bytes we can send to
the client every tick is ((20000/BitsPerSecond)/UpdateHertz) - UDPOverhead = ((20000/
8)/10) - 28 = 222 bytes. The above WARNING means that we are sending 2370 bytes to the client
in one tick. This is 10.68 times the allowed data rate.
This is usually caused by method calls to the entities located on the client i.e. entities in the AoI of the player
or the player itself. If many method calls are made, or if the method calls are passing large arguments,
then we will exceed the bandwidth allocation for the client. When bandwidth allocation is exceeded, the
position updates send to the client is reduced so entity movements will become jerky.
▪
CellApp
WARNING controller.cpp:158: Profile SCRIPT_CALL took 0.63
seconds
CellApp
WARNING Controller::standardCallback: method = onTimer; id =
17648; controllerID = 270; userArg = 260
CellApp
WARNING timer_controller.cpp:156: Profile ON_TIMER took 0.62
seconds

The above 3 WARNINGs are generated together. They mean a single onTimer() callback took 0.62 seconds.
This is very bad since it means a single entity is using 6 times a normal game tick.
▪
CellApp
WARNING TimerController::New: Rounding up initial offset to 1
from 0 (initialOffset 0.000000)

This means that you are adding a timer that is less than 0.1 seconds long. Our minimum timeout is 1 tick.

15.2. Errors
▪
CellApp
ERROR
CellApp: Scale back exhausted. spareTime 0.016686
tickPeriod 0.124365, real/entities 887/887, recvs 4504 trys 3958 good 0
errs (546 sels), nub timers 753 calls 752 rescheds

The means that the game tick has exceeded 0.1 seconds for many ticks. The CellApp has already reduced
updates to the clients to a minimum but still cannot keep the game tick shorter than 0.1 seconds.
▪
CellApp
ERROR
Chunk::load: Failed to load chunk
001xfffx/sep/0010fffco: unknown error in item 'model'

This means that there is an error loading a model in the chunk 0010fffco. Unfortunately, there is not enough
information to identify what is causing the error. You can try loading the space in World Editor and see if
the same error occurs. World Editor may be able to provide more information about the error.

123

Errors
▪
CellApp
ERROR
Received message id 41 for nonexistent entity 2844 from
192.168.50.21:49140

Message 41 is the destroyEntity message. This means that entity 2844 was already destroyed when
the CellApp received this message. Make sure that you have not called "self.destroy()" in the cell and
"self.destroyCellEntity()" in the base at the same time.
This may also be an indication of a bug in BigWorld. If you are not able to resolve the issue, please send
us more log output around the time of this error.
▪
CellApp

ERROR

DirMappingLoader::stepLoad: Stepping a loaded column

CellApp

ERROR

DirMappingLoader::stepUnload: Unloading UNLOADED column

Please ignore the above ERROR messages. They are not errors. We have removed these error messages in
BigWorld 1.8.2.
▪
BaseApp

ERROR

BaseApp::setClient: Could not find base id 2881

BaseApp

ERROR

getProxyForCall: No proxy set!

BaseApp

ERROR

getProxyForCall: No proxy set!

The above messages means that the entity (2881) has been destroyed but is still receiving messages. This
may be due to another entity calling a method using a mailbox of the destroyed entity. You have to
make sure that when an entity is destroyed, all other entities holding a mailbox to the destroyed entity is
informed so that they can reset the mailbox.
This may also be an indication of a bug in BigWorld. If you are unable to resolve the issue please send us
more logs around the time of this error.
▪
BaseApp

ERROR

0.0.0.0:0: Exception was thrown: REASON_GENERAL_NETWORK

This means that there was an error with the network. Either the machine has been disconnected from the
network, or there is some sort of hardware error.
▪
BaseApp
ERROR
10.40.3.17:56859: Exception was thrown:
REASON_NO_SUCH_PORT

This means that either the machine has been disconnected from the network, or the destination process
has crashed.

124

Chapter 16. Clock
The BigWorld server relies heavily on having an accurate clock for timers and load calculation. Unstable
clocks (clocks that may move backward) can cause undesired behaviour and even crashes. Therefore properly
configuring how BigWorld reads the time is essential for a stable, efficient server.

16.1. BigWorld Timing Methods
BigWorld provides three timing methods to select between. This can be configured by editing the file /etc/
bwmachined.conf For example, to use the gettime timing method, include a section like this:

[TimingMethod]
gettime

▪ rdtsc reads the time stamp counter register on the CPU. It is by far the fastest and least stable method
of deducing the current time. It is known to become unstable in multi-cored machines where a process
moves between cores or if frequency scaling is used. In many systems however is completely stable and
could therefore be used.
▪ gettime uses the kernel's clock driver to access the current time. It is far slower than rdtsc but has the
ability to draw on multiple time sources to ensure a good compromise between speed, accuracy and
stability is reachable. For additional system level configuration see: “Linux Clock Source” on page 125
▪ gettimeofday is a deprecated timing method. It is similar to gettime but has a lower theoretical
maximum resolution and may be disrupted by NTP.

16.2. Linux Clock Source
Linux should automatically select an appropriate clock driver for its internal timekeeping, however at times
the sysadmin may need to select a different one. If gettime is used as the timing method this will have a
direct impact to how BigWorld behaves.
A user may check which clocksource they are using with the following command run as root:
#cat /sys/devices/system/clocksource/clocksource0/current_clocksource

You may check which options available with:
#cat /sys/devices/system/clocksource/clocksource0/available_clocksource

tsc and jiffies are known to be sometimes unstable and acpi_pm will also be unstable with some
motherboard chipsets. We suggest you use hpet if it is available and if it isn't you should try acpi_pm.

125

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
Linearized                      : No
Page Count                      : 125
Profile CMM Type                : Linotronic
Profile Version                 : 2.1.0
Profile Class                   : Display Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 1998:02:09 06:49:00
Profile File Signature          : acsp
Primary Platform                : Microsoft Corporation
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : Hewlett-Packard
Device Model                    : sRGB
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 : Hewlett-Packard
Profile ID                      : 0
Profile Copyright               : Copyright (c) 1998 Hewlett-Packard Company
Profile Description             : sRGB IEC61966-2.1
Media White Point               : 0.95045 1 1.08905
Media Black Point               : 0 0 0
Red Matrix Column               : 0.43607 0.22249 0.01392
Green Matrix Column             : 0.38515 0.71687 0.09708
Blue Matrix Column              : 0.14307 0.06061 0.7141
Device Mfg Desc                 : IEC http://www.iec.ch
Device Model Desc               : IEC 61966-2.1 Default RGB colour space - sRGB
Viewing Cond Desc               : Reference Viewing Condition in IEC61966-2.1
Viewing Cond Illuminant         : 19.6445 20.3718 16.8089
Viewing Cond Surround           : 3.92889 4.07439 3.36179
Viewing Cond Illuminant Type    : D50
Luminance                       : 76.03647 80 87.12462
Measurement Observer            : CIE 1931
Measurement Backing             : 0 0 0
Measurement Geometry            : Unknown
Measurement Flare               : 0.999%
Measurement Illuminant          : D65
Technology                      : Cathode Ray Tube Display
Red Tone Reproduction Curve     : (Binary data 2060 bytes, use -b option to extract)
Green Tone Reproduction Curve   : (Binary data 2060 bytes, use -b option to extract)
Blue Tone Reproduction Curve    : (Binary data 2060 bytes, use -b option to extract)
Page Mode                       : UseOutlines
Date                            : 2008:10:16 17:12:44+11:00
PDF Version                     : 1.4
Producer                        : Apache FOP Version 0.93
Create Date                     : 2008:10:16 17:12:44+11:00

EXIF Metadata provided by EXIF.tools

Server Operations Guide

server_operations_guide

Navigation menu

Versions of this User Manual:

Views

Navigation