BDB Porting Guide

BDB-Porting-Guide

User Manual:

Open the PDF directly: View PDF PDF.
Page Count: 16

Oracle Berkeley DB
Porting Guide
12c Release 1
Library Version 12.1.6.2
Legal Notice
This documentation is distributed under an open source license. You may review the terms of this license at: http://
www.oracle.com/technetwork/database/berkeleydb/downloads/oslicense-093458.html
Oracle, Berkeley DB, and Sleepycat are trademarks or registered trademarks of Oracle. All rights to these marks are reserved. No
third-party use is permitted without the express prior written consent of Oracle.
Other names may be trademarks of their respective owners.
To obtain a copy of this document's original source code, please submit a request to the Oracle Technology Network forum at:
https://forums.oracle.com/forums/forum.jspa?forumID=271
Published 3/28/2016
3/28/2016 DB Porting Guide Page iii
Table of Contents
Preface ...................................................................................................... iv
Conventions Used in this Book .................................................................... iv
Audience .............................................................................................. iv
For More Information ............................................................................... v
Contact Us ...................................................................................... v
1. Introduction to Porting Berkeley DB ................................................................ 1
Types of Berkeley DB ports ........................................................................ 1
When Oracle Has Agreed to Support Berkeley DB on the New Platform .............. 1
When Oracle has Not Agreed to Support Berkeley DB on the New Platform ......... 2
Berkeley DB Porting Process ....................................................................... 2
2. Creating a New Berkeley DB Binary ................................................................. 3
Creating a Base Build of Berkeley DB .......................................................... 3
Determining the Scope of the Modifications .................................................. 3
Do Changes Need to be Made to the Operating System Functionality? .............. 4
Are Some Standard Functions Missing on the Target Platform? ........................ 6
How Will the Port Handle Shared Memory? ............................................... 6
What Type of Mutexes Will the Port Use? ................................................. 6
Do Any Other Changes Need to be Made? ................................................. 7
Building on the Target Platform ................................................................. 7
Source Code Layout ................................................................................. 7
3. Testing and Certifying the Port ...................................................................... 8
Types of Tests for Berkeley DB .................................................................... 8
Modifying the Tests .................................................................................. 8
Running the Tests .................................................................................... 8
Reviewing the Results of the Tests ............................................................... 9
Integrating Changes into the Berkeley DB Source Code ....................................... 9
Certifying a Port of Berkeley DB ................................................................ 10
3/28/2016 DB Porting Guide Page iv
Preface
The Berkeley DB family of open source, embeddable databases provides developers with
fast, reliable persistence with zero administration. Often deployed as "edge" databases, the
Berkeley DB family provides very high performance, reliability, scalability, and availability for
application use cases that do not require SQL.
As an open source database, Berkeley DB works on many different platforms, from Wind
River's VxWorks system, to Windows, to Android, and on most existing UNIX platforms. It runs
on 32 and 64-bit machines, little or big-endian.
Berkeley DB Porting Guide provides the information you need to port Berkeley DB 12c Release
1 (library version 12.1.6.2) to additional platforms.
Conventions Used in this Book
The following typographical conventions are used within in this manual:
Structure names are represented in monospaced font, as are method names. For example:
"DB->open() is a method on a DB handle."
Variable or non-literal text is presented in italics. For example: "Go to your DB_INSTALL
directory."
Program examples are displayed in a monospaced font on a shaded background. For example:
/* File: gettingstarted_common.h */
typedef struct stock_dbs {
DB *inventory_dbp; /* Database containing inventory information */
DB *vendor_dbp; /* Database containing vendor information */
char *db_home_dir; /* Directory containing the database files */
char *inventory_db_name; /* Name of the inventory database */
char *vendor_db_name; /* Name of the vendor database */
} STOCK_DBS;
Note
Finally, notes of interest are represented using a note block such as this.
Audience
This guide is intended for programmers porting Berkeley DB to a new platform. It assumes that
these programmers possess:
Familiarity with standard ANSI C and POSIX C 1003.1 and 1003.2 library and system calls.
Working knowledge of the target platform as well as the development tools (for example,
compilers, linkers, and debuggers) available on that platform.
3/28/2016 DB Porting Guide Page v
For More Information
Beyond this manual, you may also find the following sources of information useful when
building a DB application:
Getting Started with Berkeley DB for C
Getting Started with Transaction Processing for C
Berkeley DB Getting Started with Replicated Applications for C
Berkeley DB Installation and Build Guide
Berkeley DB Programmer's Reference Guide
Berkeley DB Getting Started with the SQL APIs
Berkeley DB C API Reference Guide
To download the latest Berkeley DB documentation along with white papers and other
collateral, visit http://www.oracle.com/technetwork/indexes/documentation/index.html.
For the latest version of the Oracle Berkeley DB downloads, visit http://www.oracle.com/
technetwork/database/database-technologies/berkeleydb/downloads/index.html.
Contact Us
You can post your comments and questions at the Oracle Technology (OTN) forum for Oracle
Berkeley DB at: https://forums.oracle.com/forums/forum.jspa?forumID=271, or for Oracle
Berkeley DB High Availability at: https://forums.oracle.com/forums/forum.jspa?forumID=272.
For sales or support information, email to: berkeleydb-info_us@oracle.com You can subscribe
to a low-volume email announcement list for the Berkeley DB product family by sending email
to: bdb-join@oss.oracle.com
3/28/2016 DB Porting Guide Page 1
Chapter 1. Introduction to Porting Berkeley DB
Berkeley DB is an open source database product that supports a variety of platforms. When
there is a need to run Berkeley DB on a platform that is currently not supported, DB is
distributed in source code form that you can use as base source to port Berkeley DB to that
platform.
Berkeley DB is designed to be as portable as possible, and has been ported to a wide variety
of systems, from Wind River's Tornado system, to VMS, to Windows/NT and Windows/95,
and most existing UNIX platforms. It runs on 16-bit, 32-bit, and 64-bit machines, little or
big-endian. The difficulty of a port depends on how much of the ANSI C and POSIX 1003.1
standards the new architecture offers.
Before you begin actually porting Berkeley DB, you need an understanding of the:
Types of Berkeley DB ports (page 1)
Berkeley DB Porting Process (page 2)
Types of Berkeley DB ports
There are several types of Berkeley DB ports:
Ports developed and supported by Oracle
Ports developed by a customer or a partner, but which Oracle has agreed to support.
Ports developed, maintained, and supported by a customer or partner.
For a port developed by a customer or a partner, the general steps for porting Berkeley DB
to a new platform are the same whether or not Oracle has agreed to support Berkeley DB on
the new platform. For example, after you complete the port you send it to Berkeley DB as
described in Integrating Changes into the Berkeley DB Source Code (page 9). However,
there are some differences.
When Oracle Has Agreed to Support Berkeley DB on the New Platform
When porting Berkeley DB to a platform that Oracle has agreed to support, you need to have
Berkeley DB engineering review your port at various points. These review points are discussed
more fully in Integrating Changes into the Berkeley DB Source Code (page 9), Modifying the
Tests (page 8), and Reviewing the Results of the Tests (page 9).
It is up to you to submit the results of the tests (test_micro, test_mutex, and, if possible,
the entire tcl test suite) for review by Oracle Berkeley DB engineering in order for Oracle to
consider providing support for Berkeley DB on a new platform.
You must also assign copyrights for your changes to any part of Berkeley DB to "Oracle
Corporation" and attest to the fact that you are not infringing on any software patents for the
changes to be included in the general Berkeley DB distribution.
Library Version 12.1.6.2 Introduction to Porting Berkeley DB
3/28/2016 DB Porting Guide Page 2
Once the port is certified, Oracle provides support for Berkeley DB on the new platform in the
same manner that it does for Berkeley DB running on other established platforms.
When Oracle has Not Agreed to Support Berkeley DB on the New Platform
When Oracle has not agreed to support Berkeley DB on the new platform, the customer or
partner assume the responsibility of front-line support. When it is determined that there is
a problem in the code that was not modified by the customer or partner, then Berkeley DB
engineering provides support to the customer or vendor who implemented the port, However,
in these cases, Oracle needs access to the platform and hardware for diagnosing, debugging,
and testing.
Berkeley DB Porting Process
As with any porting project, porting Berkeley DB to a new platform consists of the following
process:
1. Determine the modifications that you need to make in the base code and create a
working executable of Berkeley DB on the target platform as described in Creating a New
Berkeley DB Binary (page 3).
2. Perform final test and quality assurance functions on the target platform as described in
Testing and Certifying the Port (page 8).
3/28/2016 DB Porting Guide Page 3
Chapter 2. Creating a New Berkeley DB Binary
Creating a new Berkeley DB executable on the target platform, involves:
1. Creating a Base Build of Berkeley DB (page 3)
2. Determining the Scope of the Modifications (page 3)
3. Building on the Target Platform (page 7)
Creating a Base Build of Berkeley DB
The simplest way to begin a port is to attempt to configure and build Berkeley DB on a UNIX or
UNIX-like system. This gives you a list of the files that you needed to build Berkeley DB as well
as the configuration files you can use as a starting point for building on your target port.
To create a base build of Berkeley DB, following the instructions in the Berkeley DB
Programmer's Reference Guide:
1. Download a Berkeley DB distribution from http://www.oracle.com/technetwork/
database/database-technologies/berkeleydb/downloads/index.html.
2. Build Berkeley DB.
Berkeley DB uses the GNU autoconf tools for configuration on almost all of the platforms it
supports. Specifically, the include file db_config.h configures the Berkeley DB build. The
simplest way to begin a port is to configure and build Berkeley DB on a UNIX or UNIX-like
system, and then take the Makefile and db_config.h file created by that configuration,
and modify it by hand to reflect the needs of the new architecture. Unless you are already
familiar with the GNU autoconf toolset, we do not recommend you take the time to integrate
your changes back into the Berkeley DB autoconfiguration framework. Instead, send us context
diffs of your changes and any new source files you created, and we can integrate the changes
into our source tree.
Determining the Scope of the Modifications
Once you have a good build of Berkeley DB on a UNIX or UNIX-like system, look over the code
to determine what type of code changes you need to make so that you can successfully build
Berkeley DB on your target system. This process involves determining:
Do Changes Need to be Made to the Operating System Functionality? (page 4)
Are Some Standard Functions Missing on the Target Platform? (page 6)
How Will the Port Handle Shared Memory? (page 6)
What Type of Mutexes Will the Port Use? (page 6)
Do Any Other Changes Need to be Made? (page 7)
Library Version 12.1.6.2 Creating a New Berkeley DB Binary
3/28/2016 DB Porting Guide Page 4
Do Changes Need to be Made to the Operating System Functionality?
Berkeley DB uses about forty operating system primitives. The Berkeley DB distribution
contains files which are wrappers around these operating system primitives that act as
an abstraction layer to separate the main Berkeley DB code from operating system and
architecture-specific components. You must port these files (or versions of these files)
whenever you port Berkeley DB to a new platform.
Within a Berkeley DB distribution, typically, there is only a single version of these files for
all platforms that Berkeley DB supports. Those versions of the files live in the os directory
of the distribution and follow the ANSI C and POSIX 1003.1 standards. Within each file, there
is usually one, but sometimes several functions (for example, the os_alloc.c file contains
functions such as malloc, realloc, strdup, free). The following table describes the files in
the os directory of the Berkeley DB distribution along with the POSIX functions that must be
ported.
POSIX Functions Internal Function Name Source File
abort() is required if diagnostic build
is used or if snprintf is not provided
by the platform
__os_abort() os_abort.c
freeaddrinfo()
getaddrinfo(), htonl(), htons(),
inet_addr(), and gethostbyname() are
required for Replication Manager
__os_getaddrinfo(),
__os_freeaddrinfo()
os_addrinfo.c
malloc(), realloc(), strdup(), free(),
memcpy(), memset(), strlen()
__os_umalloc(), __os_urealloc(),
__os_ufree(), __os_strdup(),
__os_calloc(), __os_malloc(),
__os_realloc(), __os_free(),
__os_guard(), __ua_memcpy()
os_alloc.c
clock_gettime(), time(),
gettimeofday()
__os_gettime() os_clock.c
sysconf() __os_cpu_count() os_cpu.c
ctime(), ctime_r() __os_ctime() os_ctime.c
opendir(), closdir(), readdir(), stat() __os_dirlist(), __os_dirfree() os_dir.c
strncpy() __os_get_errno_ret_zero(),
__os_get_errno(), __os_get_syserr(),
__os_set_errno(), __os_strerror(),
__os_posix_err()
os_errno.c
fcntl() is required for DB_REGISTER __os_fdlock() os_flock.c
fsync(), fdatasync() __vx_fsync(), __os_fsync() os_fsync.c
getenv() and strcpy() are required
when environment variables are used
to configure the database
__os_getenv() os_getenv.c
Library Version 12.1.6.2 Creating a New Berkeley DB Binary
3/28/2016 DB Porting Guide Page 5
POSIX Functions Internal Function Name Source File
close(), open() __os_openhandle(),
__os_closehandle()
os_handle.c
getpid()
pthread_self() is required for
replication and failchk functionality
__os_id() os_pid.c
shmget(), shmdt(), shmctl(),
and shmat() are required when
envrionment uses share memory for
regions
munmap() is required when
envrionment uses memory mapped
files for regions or read-only
databases
munlock() is required when
environment is configured with
DB_LOCKDOWN
__os_attach(), __os_detach(),
__os_mapfile(), __os_unmapfile(),
__os_map(), __shm_mode(),
__no_system_mem()
os_map.c
mkdir(), chmod() __os_mkdir() os_mkdir.c
fchmod()
directio() is required when explicitly
enabling DIRECTIO_ON
__os_open() os_open.c
rename() __os_rename() os_rename.c
getuid() is required when
environment variables are used to
configure the database
__os_isroot() os_root.c
read(), write(), pread(), pwrite() __os_io(), __os_read(), __os_write(),
__os_physwrite()
os_rw.c
lseek() __os_seek() os_seek.c
stat(), fstat() __os_exists(), __os_ioinfo() os_stat.c
ftruncate() is required when using
truncate
__os_truncate() os_truncate.c
unlink() __os_unlink() os_unlink.c
yield(), sched_yield() __os_yield(), __os_sleep() os.yield.c
When the operating system primitives on the target platform are identical or close to
the POSIX semantics that Berkeley DB requires, then no code changes or minimal code
changes to the files in the os directory are required. If the operating system primitives are
quite different, then some code changes may be required to bridge the gap between the
requirements of Berkeley DB and what the operating system provides.
Library Version 12.1.6.2 Creating a New Berkeley DB Binary
3/28/2016 DB Porting Guide Page 6
Where different code is required, you write an entirely different version of the file and
place it in an os_xxx directory where xxx represents a platform name. There are os_xxx
subdirectories in the Berkeley DB distribution for several established non-POSIX platforms.
For example, there is a os_vxworks directory that contains VxWorks versions of some of the
files in the os directory, and Windows versions of some files are in the os_windows directory.
If your target platform needs a different version of a file, you will need to write that file and
place it in a new os_xxx directory that you create for your target platform.
Are Some Standard Functions Missing on the Target Platform?
In some cases, the target platform may not provide the few POSIX functions required by
Berkeley DB or the functions provided by the target platform may not operate in a standard
compliant way. Berkeley DB provides replacement functions in the clib directory of the
Berkeley DB distribution.
You need to determine how your target platfrom handles these functions:
When the target platform does not have a POSIX function required by Berkeley DB, no
action is required on your part. When Berkeley DB cannot find one of these functions on
the target platform, it automatically uses the replacement functions supplied in the clib
directory of the Berkeley DB distribution. For example, if the target platform does not have
the atoi or strtol functions, Berkeley DB uses clib/atoi.c and clib/strtol.c.
When the target platform has a function required by Berkeley DB, but that function
operates in a non-standard compliant way, you can code to the replacement functions
supplied in the clib directory.
How Will the Port Handle Shared Memory?
In order to write multiprocess database applications (not multithreaded, but threads of
control running in different address spaces), Berkeley DB must be able to name pieces of
shared memory and access them from multiple processes.
On UNIX/POSIX systems, Berkeley DB uses mmap and shmget for that purpose, but any
interface that provides access to named shared memory is sufficient. If you have a simple,
flat address space, you should be able to use the code in os_vxworks/os_map.c as a starting
point for the port.
If you are not intending to write multiprocess database applications, then this won't be
necessary, as Berkeley DB can simply allocate memory from the heap if all threads of control
will live in a single address space.
What Type of Mutexes Will the Port Use?
Berkeley DB requires some form of self-blocking mutual exclusion mutex. Blocking mutexes
are preferred as they tend to be less CPU-expensive and less likely to cause thrashing.
If blocking mutexes are not available, however, test-and-set will work as well. The code
for mutexes is in two places in the system: the include file dbinc/mutex_int.h, and the
distribution directory mutex.
Library Version 12.1.6.2 Creating a New Berkeley DB Binary
3/28/2016 DB Porting Guide Page 7
Do Any Other Changes Need to be Made?
In most cases, you do not need to make any changes to the Berkeley DB source code that
is not in the abstraction layer (that is, in the os directory) as that code is designed to be
platform-independent code. However, in some situations, the compiler for the target platform
is non-standard and may raise errors when compiling some aspects of the Berkeley DB code
(for example, additional casting may be required, or a certain type may cause a problem). In
these cases, you will need to modify the generic Berkeley DB code in order to have error-free
compilation.
Building on the Target Platform
Once you have an idea of the scope of the modifications, use the files generated on the UNIX
system to help you create any compiler or linker tool chain files that you need to build on the
target platform. At this point, start building the Berkeley DB on the target platform making
the changes to the code that you identified earlier in the process. Once you have identified
the modifications that you need to make, change the code accordingly.
Source Code Layout
The following table describes the directories in the Berkeley DB distribution.
Directory Description
LICENSE Berkeley DB License
build_android Android build directory
build_unix UNIX build directory
build_vxworks VxWorks build directory
build_wince Windows CE build directory
build_windows Windows build directory
dist Scripts used to auto-generate code for
Berkeley DB distribution and administration
docs Documentation
examples Example programs for various language APIs
lang Implementation of various APIs that work with
the Berkeley DB library
src Implementation of the Berkeley DB library
test Test suites
util Implementation of utilities that can be used
with Berkeley DB
3/28/2016 DB Porting Guide Page 8
Chapter 3. Testing and Certifying the Port
There are several different types of tests available for validating your port of Berkeley DB as
discussed in Types of Tests for Berkeley DB (page 8). Testing your port involves:
Modifying the Tests (page 8)
Running the Tests (page 8)
Reviewing the Results of the Tests (page 9)
Integrating Changes into the Berkeley DB Source Code (page 9)
Certifying a Port of Berkeley DB (page 10)
Types of Tests for Berkeley DB
There are two types of tests available for testing your port of Berkeley DB:
The C Tests for Berkeley DB
There are two types of C tests for Berkeley DB. Each of these is in its own directory:
The test_mutex test is found in src/mutex.
The test_micro tests are found in test/micro. Here you will find C tests that exercise
the most common code paths. However, please note that test_micro is not intended to
be an exhaustive test suite. Additionally, it tests the different versions of Berkeley DB
(including the new port) against each other. The test_micro tests can either be run in a
shell or as simple C tests.
The Berkeley DB Test Suite
The test directory contains the Berkeley DB Test Suite that tests all of the code in Berkeley
DB. Using the Test Suite involves using Tool Command Language (Tcl) version 8.5 or later.
Running the standard version of the Test Suite executes tests the major functionality of
Berkeley DB. A more exhaustive version of the Test Suite runs all the tests several more
times, testing encryption, replication, and different page sizes.
Modifying the Tests
There should be no need to make modifications to the tests. However, in a few situations,
the test hardware may have some constraints (for example, small amount of memory) which
may cause certain tests (which expect more memory) to fail. In these rare situations, you may
need to modify the tests to work within the constraints of the platform.
When Oracle has agreed to support Berkeley DB on the new platform, submit any proposed
test changes for review and approval by Oracle Engineering before running the tests.
Running the Tests
You test your new port of Berkeley DB by running the tests in the following order:
Library Version 12.1.6.2 Testing and Certifying the Port
3/28/2016 DB Porting Guide Page 9
1. Run the C tests in the following order:
a. Tests for mutexes located in the src/mutex directory. Build and run the tests in your
build directory:
% make test_mutex
% ./test_mutex
b. Tests for the common code paths located in the test/micro directory. To run the
tests in a shell script, follow the instructions in the test/micro/README file.
2. If the target platform supports the use of Tcl (version 8.5 or later), run the Test Suite.
How you run the Test Suite varies depending on the target platform:
If the target platform supports a UNIX-like version of Tcl, then set up Tcl and build
the Test Suite as described in "Running the Test Suite under UNIX" in Berkeley DB
Installation and Build Guide at http://docs.oracle.com/cd/E17076_02/html/
installation/build_unix_test.html and, then, run the test suite.
If the target platform supports a Windows-like version of Tcl, then setup Tcl, and
build and run the Test Suite as described in "Running the Test Suite under Windows" in
Berkeley DB Programmer's Reference Guide at http://docs.oracle.com/cd/E17076_02/
html/installation/build_win_test.html
Reviewing the Results of the Tests
It is up to you to submit the results of the tests (test_micro, test_mutex, and, if possible,
the entire tcl test suite) for review by Oracle Berkeley DB engineering in order for Oracle to
consider providing support for Berkeley DB on a new platform.
When Oracle has not agreed to support Berkeley DB on the new platform, you are responsible
for ensuring that the tests run successfully.
Integrating Changes into the Berkeley DB Source Code
Once you have have completed your port, send all code changes that you made as "diff" files
to Berkeley DB development for review.
Exactly how you integrate changes into the Berkeley DB source code varies depending on
whether or not Oracle has agreed to support Berkeley DB on the new platform:
At this point, when Oracle has agreed to support Berkeley DB on the new platform:
1. Berkeley DB development reviews the changes, makes the changes (with modifications if
necessary) in the source code, creates another snapshot which includes the new platform-
specific changes, and sends you the new snapshot.
2. On receipt of the new snapshot, recompile this new snapshot. If the new snapshot
compiles correctly without any changes, test the port again.
Library Version 12.1.6.2 Testing and Certifying the Port
3/28/2016 DB Porting Guide Page 10
Certifying a Port of Berkeley DB
When the target platform supports using Tcl, the port is considered certified after a successful
standard run (run_std) of the Test Suite .
When the target platform does not support using Tcl, a port is considered certified if you see
successful message reports for each of the tests in test_micro and test_mutex.
Additionally, the configuration and compilation of Berkeley DB must be complete without
errors or warninigs of any kind. You must provide specific information about the hardware,
software, and compiler used during the porting process, especially versions of all thrid party
tools used. If you have other diagnostic tools available, such as memory allocation checking
tools, please conduct tests using them as well and provide Oracle engineering with the results.
Finally, do not constrain your testing to one configuration. There are many different ways
to configure Berkeley DB (that is many options to the "configure" script), please configure,
built, and test Berkeley DB on the target platform with as many combinations of these flags as
possible to ensure that nothing is missed.
Note
Contact the Oracle Berkeley DB engineering team for a platform compatibility test
suite.

Navigation menu